Build docs with fresh CPython commit not ~24h old one

[hugovk]

A full build of all languages / versions has been taking somewhere between 24 and 50 hours (#169, but python/cpython#123113 should cut about a third).

We update the CPython repo once at the start, then loop each language/version combo:

docsbuild-scripts/build_docs.py

Lines 1120 to 1135 in 56d72d4

	cpython_repo.update()
	while todo:
	version, language = todo.pop()
	logging.root.handlers[0].setFormatter(
	logging.Formatter(
	f"%(asctime)s %(levelname)s {language.tag}/{version.name}: %(message)s"
	)
	)
	if sentry_sdk:
	with sentry_sdk.configure_scope() as scope:
	scope.set_tag("version", version.name)
	scope.set_tag("language", language.tag)
	builder = DocBuilder(
	version, versions, language, languages, cpython_repo, **vars(args)
	)
	all_built_successfully &= builder.run()

This means builds near the end of the loop will be using a Git commit which could be a day or two old.

For example, looking at the current logs:

15502:2024-08-24 16:07:01,574 DEBUG: Run: 'git -C /srv/docsbuild/cpython fetch'

It's currently 2024-08-25 10:45, meaning current builds are using an 18-hour-old commit, and we're about half way through a full build.

So let's instead update the CPython repo before each language/version, perhaps by moving the update inside the while loop:

     cpython_repo = Repository(
         "https://github.com/python/cpython.git", args.build_root / "cpython"
     )
-    cpython_repo.update()
     while todo:
         version, language = todo.pop()
         logging.root.handlers[0].setFormatter(
                 f"%(asctime)s %(levelname)s {language.tag}/{version.name}: %(message)s"
             )
         )
         if sentry_sdk:
             with sentry_sdk.configure_scope() as scope:
                 scope.set_tag("version", version.name)
                 scope.set_tag("language", language.tag)
+        cpython_repo.update()
         builder = DocBuilder(
             version, versions, language, languages, cpython_repo, **vars(args)
         )

[ned-deily]

It might also make sense to reverse the order of builds so that the newest releases (where there is the most immediate interest and churn) will be built first.

[hugovk]

They're already being run in reverse:

diff --git a/build_docs.py b/build_docs.py
index 93dcac4..1756c2b 100755
--- a/build_docs.py
+++ b/build_docs.py
@@ -1117,7 +1117,7 @@ def build_docs(args) -> bool:
     cpython_repo = Repository(
         "https://github.com/python/cpython.git", args.build_root / "cpython"
     )
-    cpython_repo.update()
+    # cpython_repo.update()
     while todo:
         version, language = todo.pop()
         logging.root.handlers[0].setFormatter(
@@ -1125,6 +1125,8 @@ def build_docs(args) -> bool:
                 f"%(asctime)s %(levelname)s {language.tag}/{version.name}: %(message)s"
             )
         )
+        print(f"{version.name}/{language.tag}")
+        continue
         if sentry_sdk:
             with sentry_sdk.configure_scope() as scope:
                 scope.set_tag("version", version.name)
@@ -1136,6 +1138,7 @@ def build_docs(args) -> bool:
     logging.root.handlers[0].setFormatter(
         logging.Formatter("%(asctime)s %(levelname)s: %(message)s")
     )
+    sys.exit()
 
     build_sitemap(versions, languages, args.www_root, args.group)
     build_404(args.www_root, args.group)

3.14/zh-tw
3.14/zh-cn
3.14/uk
3.14/tr
3.14/pt-br
3.14/pl
3.14/ko
3.14/ja
3.14/it
3.14/id
3.14/fr
3.14/es
3.14/en
3.13/zh-tw
3.13/zh-cn
3.13/uk
3.13/tr
3.13/pt-br
3.13/pl
3.13/ko
3.13/ja
3.13/it
3.13/id
3.13/fr
3.13/es
3.13/en
3.12/zh-tw
3.12/zh-cn
3.12/uk
3.12/tr
3.12/pt-br
3.12/pl
3.12/ko
3.12/ja
3.12/it
3.12/id
3.12/fr
3.12/es
3.12/en

It's because we sort the versions from lowest to highest, but then pop()s from the list.