From a6d4d81ee257e7ae217a39d1d1bcdd5a948919fa Mon Sep 17 00:00:00 2001 From: Clark Boylan Date: Wed, 20 Oct 2021 10:12:43 -0700 Subject: [PATCH] Add zuul debugging documentation Try to provide some hints for debugging unexpected zuul behaviors in our documentation. Change-Id: I937df9d8d71ae4984991aa9f327dd1351d539284 --- doc/source/zuul.rst | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst index 643360d49c..08d2a63c2e 100644 --- a/doc/source/zuul.rst +++ b/doc/source/zuul.rst @@ -228,6 +228,39 @@ YAML, there is no need to recombine split lines):: .. _zuul_github_projects: +Debugging Problems +------------------ + +Occasionally you'll have a job enter an error state or an entire change that +appears to be stuck in a Zuul pipeline. Debugging these problems can be a bit +daunting to start as Zuul's logs are quite verbose. The good news is that once +you learn a few tricks those verbose logs become quite the powerful tool. + +Often the best place to start is grepping the Zuul scheduler debug log for +the pipeline entry identifier (eg change number, tag, or rev sha1):: + + you@zuul02$ grep 123456 /var/log/zuul/debug.log + you@zuul02$ grep c6229660cda0af42ecd5afbe7fefdb51136a0436 /var/log/zuul/debug.log + +In many of these log lines you'll see Zuul event IDs like +`[e: 1718628fe39643e1bd6a88a9a1477b4f]` this ID identifies the event that +triggered Zuul to take action for these changes and is logged through all +the Zuul services. It can be very powerful to do a grep on this event ID and +trace through the actions that the scheduler took for this event:: + + you@zuul02$ grep 1718628fe39643e1bd6a88a9a1477b4f /var/log/zuul/debug.log + +This might lead you to look at executor logs where you can use the same +ID to grep for actions related to this even on the executor:: + + you@ze01$ grep 1718628fe39643e1bd6a88a9a1477b4f /var/log/zuul/executor-debug.log + +As you trace through the logs related to a change or event ID you can look for +`ERROR` or `Traceback` messages to try and identify the underlying source of +the problem. Note that `Traceback` messages are not prefixed with the event +ID which means you'll have to grep with additional context, for example using +`grep -B20 -A20`. + GitHub Projects ===============