I had run celery for my project using 10 threads. It worked perfectly. Now, when i use the command:
celery - A tasks worker--loglevel = info--concurrency 10
celery get stuck at :
[2014 - 08 - 07 12: 55: 21, 013: INFO / MainProcess] Connected to amqp: //guest:**@127.0.0.1:5672//
[2014 - 08 - 07 12: 55: 21, 025: INFO / MainProcess] mingle: searching
for neighborsand when i try to end this using ctrl+c, it doesn't shutdown.
[2014 - 08 - 07 12: 55: 21, 013: INFO / MainProcess] Connected to amqp: //guest:**@127.0.0.1:5672//
[2014 - 08 - 07 12: 55: 21, 025: INFO / MainProcess] mingle: searching
for neighbors
^
C
worker: Hitting Ctrl + C again will terminate all running tasks!
worker: Warm shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess) ^
C
worker: Cold shutdown(MainProcess)It should also be noted that if I add --autoscale=50,1 to celery, when it gets "stuck" (I really wish I had a better term for this), celery workers just accumulate until all 50 workers are present, and it still doesn't process any tasks.,.save(), for some reason, hangs. I shoved some print statements in there after every line and the one after .save() does not print. Maybe it's a race condition of sorts? Table locking, perhaps?,When I comment out its contents and put logger.warning("test") in it instead, the celery tasks seem to finish and the queue doesn't get "stuck." I don't think it's because of the chain, however, because if I replace the task's contents with:,watering_seconds and timeout_seconds will be user-adjustable, and it wouldn't be unreasonable for them to be set to several minutes. Is there a way I can make this work while leaving the Beat interval at three seconds?
Here is my Celery "tasks" file and my Django project's settings.py, which has this Celery Beat configuration (the timedelta is set to every 3 seconds in a user-editable configuration file):
CELERYBEAT_SCHEDULE = {
"UpdateEnabledSensors": {
"task": "pidrator.hardware_controller.UpdateEnabledSensors",
"schedule": timedelta(seconds = PIDRATOR_SENSOR_UPDATE_FREQUENCY)
}
}I think the problem is in the WateringCycle task, which looks like this:
@celery_pidrator.task
def WateringCycle(irrigator):
from celery
import chain
chain(
SetWateringCycle.si(irrigator, True),
ActuateRelay.si(irrigator.relay, True),
ActuateRelay.subtask(
(irrigator.relay, False),
countdown = irrigator.watering_seconds,
immutable = True),
SetWateringCycle.subtask(
(irrigator, False),
countdown = irrigator.timeout_seconds,
immutable = True)
).apply_async()When I comment out its contents and put logger.warning("test") in it instead, the celery tasks seem to finish and the queue doesn't get "stuck." I don't think it's because of the chain, however, because if I replace the task's contents with:
@celery_pidrator.task
def WateringCycle(irrigator):
SetWateringCycle(irrigator, True)
ActuateRelay(irrigator.relay, True)
ActuateRelay.apply_async(
(irrigator.relay, False),
countdown = irrigator.watering_seconds)
SetWateringCycle.apply_async(
(irrigator, False),
countdown = irrigator.watering_seconds + irrigator.timeout_seconds)Okay, I think I narrowed it down to SetWateringCycle.
@celery_pidrator.task
def SetWateringCycle(irrigator, in_watering_cycle):
irrigator.in_watering_cycle = in_watering_cycle
irrigator.save(update_fields = ["in_watering_cycle"])
logger.warning(
"Watering cycle %s for \"%s\".",
("finished", "started")[in_watering_cycle], irrigator.name)a builder is working along and then tries to take a task: Dec 31 00:46:17 buildhw-03.phx2.fedoraproject.org kojid[1982746]: 2019-12-31 00:46:17,562 [INFO] {1982746} ko ji.TaskManager:1210 Attempting to take task 40002458, a builder is working along and then tries to take a task: Dec 31 00:46:17 buildhw-03.phx2.fedoraproject.org kojid[1982746]: 2019-12-31 00:46:17,562 [INFO] {1982746} ko ji.TaskManager:1210 Attempting to take task 40002458 ,https://bodhi.fedoraproject.org/releases/F32,@cverna can you fix https://bodhi.fedoraproject.org/updates/FEDORA-2019-9184b1e049 please?
[root @rabbitmq01~][PROD] # rabbitmqctl list_queues - p "/pubsub"
Listing queues
amqp_to_zmq 386
faf 10697
openqa.stg_scheduler 0
openqa_relvalamiconsumer.test 6438
federation: amq.topic - > rabbit @rabbitmq01.phx2.fedoraproject.org: /public_pubsub:amq.topic 0
robosignatory 1
fedora - build - checks 0
greenwave 0
wiki 577508[root @rabbitmq03~][PROD] # rabbitmqctl list_queues - p / pubsub--online
Listing queues
openqa_checkcomp 0
openqa_relvalamiconsumer.test 6438
bodhi_composer 0
openqa_wiki_reporter 0
centos - ci 1029
bodhi 0
openqa_autocloudreporter.stg 0
amqp_bridge_verify_missing 19
federation: zmq.topic - > rabbit @rabbitmq01.phx2.fedoraproject.org: /public_pubsub:zmq.topic 0
mts 0
the - new - hotness 55
robosignatory 0
openqa_scheduler 0
openqa_resultsdb_reporter 0
openqa_autocloudreporter 0
openqa.stg_scheduler 0
faf 10707
amqp_to_zmq 0
federation: amq.topic - > rabbit @rabbitmq01.phx2.fedoraproject.org: /public_pubsub:amq.topic 0
resultsdb_ci_listener 0
wiki 578159
openqa_relvalamiconsumer 0
openqa_relvalconsumer.test 0
fedora - build - checks 0
koschei 1
openqa_relvalconsumer 0
coreos 0
greenwave 38...snip a bunch more of the same... | bodhi2 | | | ExclusiveLock | t | virtualxid | bodhi2 | 2019 - 12 - 29 01: 33: 07.221469 + 00 | 00: 48: 26.857518 | 21308 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180412 bodhi2 | | 2721427923 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 01: 24: 07.640285 + 00 | 00: 57: 26.438702 | 13713 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180397 bodhi2 | | 2721388468 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 00: 54: 06.819603 + 00 | 01: 27: 27.259384 | 18230 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180386 bodhi2 | | 2721428424 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 01: 39: 07.720449 + 00 | 00: 42: 26.358538 | 26261 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180414 bodhi2 | | 2721428952 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 01: 54: 07.422068 + 00 | 00: 27: 26.656919 | 6601 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180421 bodhi2 | | 2721411721 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 01: 06: 08.436176 + 00 | 01: 15: 25.642811 | 29656 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180391 bodhi2 | | 2721370917 | ExclusiveLock | t | transactionid | bodhi2 | 2019 - 12 - 29 00: 45: 06.902888 + 00 | 01: 36: 27.176099 | 9914 | UPDATE comments SET update_id = 176147, user_id = 91 WHERE comments.id = 1180383(226 rows)
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,073 [INFO] {530977} koji.TaskManager:1210 Attempting to take task 40025476
...
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koji:2580 url: https://koji.fedoraproject.org/kojihub?session-id=92635914&session-key=<redacted>&callnum=1093
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koj i:2589 headers: {'User-Agent': 'koji/1', 'Content-Type': 'text/xml', 'Content-Length': '159'}
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koj i:2589 data: "<?xml version='1.0'?>\n<methodCall>\n<methodName>host.openTask</methodName>\n<params>\n<param>\n<value><int>40025476</int></value>\n</param>\n</params>\n</methodCall>\n"
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koji:2589 stream: True
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koji:2589 verify: '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'
Jan 01 06:44:28 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 06:44:28,087 [DEBUG] {530977} koji:2589 timeout: 43200
Jan 01 09:38:02 buildhw-10.phx2.fedoraproject.org kojid[530977]: 2020-01-01 09:38:02,460 [DEBUG] {535641} koji:2213 Opening new requests session
[Wed Jan 01 06: 44: 28.106504 2020][: error][pid 5258] 2020 - 01 - 01 06: 44: 28, 106[INFO] m = host.openTask u = buildhw - 10. phx2.fedoraproject.org p = 5258 r = 10.5 .126 .9: 33346 koji.xmlrpc: Completed method host.openTask
for session 92635914(#1093): 0.010418 seconds, rss 36736, stime 0.362359'GitBox Fri, 16 Apr 2021 11:07:33 -0700
minnieshi edited a comment on issue #13542:
URL: https://github.com/apache/airflow/issues/13542# issuecomment - 821356514"Well," said Miss Dorcas, reflectively, "I always said it doesn't do to carry exclusiveness too far. Poor dear Papa was quite a democrat. He often said that he had[Pg 9] seen quite good manners and real refinement in people of the most ordinary origin.","We'll call upon her when they get well settled," said Miss Dorcas, definitively.,"Well, then, do take a little pains to say what you mean," said Miss Dorcas.,"And ready to go to the devil straight off, I suppose?" said the Doctor. "His doors are always open."
The Project Gutenberg EBook of We and Our Neighbors, by Harriet Beecher Stowe This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever.You may copy it, give it away or re - use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org / license Title: We and Our Neighbors or, The Records of an Unfashionable Street Author: Harriet Beecher Stowe Illustrator: Alfred Fredericks Release Date: March 29, 2015[EBook #48603] Language: English Character set encoding: ISO-8859-1 *** START OF THIS PROJECT GUTENBERG EBOOK WE AND OUR NEIGHBORS *** Produced by David Edwards, Diane Monico, and the Online Distributed Proofreading Team at http://www.pgdp.net (This file was produced from images generously made available by The Internet Archive)