[recipes] Add roll_deps to gclient

roll_deps method in gclient recipe_module allow users to pass DEPS file together with recipes that need to be rolled. It returns a list of files that need to be edited in addition to the modified DEPS file (replaced inline). R=jojwang, keybo Change-Id: I6987849f0ebacbb063d6f6186af1dba1647550a7 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4785897 Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Xinan Lin <linxinan@chromium.org>
2 years ago · 2a951b8648
parent 45091ca1c8
commit 2a951b8648
7 changed files with 260 additions and 20 deletions
--- a/recipes/README.recipes.md
+++ b/recipes/README.recipes.md
@ -161,7 +161,7 @@ Returns (Path): The "depot_tools" root directory.
 &emsp; **@property**<br>&mdash; **def [upload\_to\_google\_storage\_path](/recipes/recipe_modules/depot_tools/api.py#21)(self):**
 ### *recipe_modules* / [gclient](/recipes/recipe_modules/gclient)

-[DEPS](/recipes/recipe_modules/gclient/__init__.py#1): [git](#recipe_modules-git), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+[DEPS](/recipes/recipe_modules/gclient/__init__.py#1): [git](#recipe_modules-git), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]


 #### **class [GclientApi](/recipes/recipe_modules/gclient/api.py#77)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
@ -228,6 +228,24 @@ Args:

 &mdash; **def [resolve\_revision](/recipes/recipe_modules/gclient/api.py#161)(self, revision):**

+&mdash; **def [roll\_deps](/recipes/recipe_modules/gclient/api.py#428)(self, deps_path, dep_updates, test_data=None):**
+
+Updates DEPS file to desired revisions, and returns all requried file
+changes.
+
+Args:
+  deps_path - Path to DEPS file that will be modified.
+  dep_updates - A map of dependencies to update (key = dependency name,
+                value = revision).
+
+Returns:
+  A map of all files that need to be modified (key = file path, value = file
+  content) in addition to DEPS file itself.
+  Note: that git submodules (gitlinks) are treated as files and content is a
+  commit hash.
+  Note: deps_path is not added to returned map since the repo relative path
+  is not known.
+
 &mdash; **def [runhooks](/recipes/recipe_modules/gclient/api.py#285)(self, args=None, name='runhooks', \*\*kwargs):**

 &mdash; **def [set\_patch\_repo\_revision](/recipes/recipe_modules/gclient/api.py#355)(self, gclient_config=None):**
@ -1038,10 +1056,10 @@ Raises:
 &mdash; **def [RunSteps](/recipes/recipe_modules/depot_tools/examples/full.py#16)(api):**
 ### *recipes* / [gclient:examples/full](/recipes/recipe_modules/gclient/examples/full.py)

-[DEPS](/recipes/recipe_modules/gclient/examples/full.py#7): [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+[DEPS](/recipes/recipe_modules/gclient/examples/full.py#7): [gclient](#recipe_modules-gclient), [recipe\_engine/assertions][recipe_engine/recipe_modules/assertions], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]


-&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/examples/full.py#68)(api):**
+&mdash; **def [RunSteps](/recipes/recipe_modules/gclient/examples/full.py#74)(api):**
 ### *recipes* / [gclient:tests/diff\_deps](/recipes/recipe_modules/gclient/tests/diff_deps.py)

 [DEPS](/recipes/recipe_modules/gclient/tests/diff_deps.py#10): [gclient](#recipe_modules-gclient), [recipe\_engine/assertions][recipe_engine/recipe_modules/assertions], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io]
--- a/recipes/recipe_modules/gclient/init.py
+++ b/recipes/recipe_modules/gclient/init.py
@ -1,15 +1,16 @@
 DEPS = [
-  'git',
-  'gitiles',
-  'recipe_engine/buildbucket',
-  'recipe_engine/context',
-  'recipe_engine/json',
-  'recipe_engine/path',
-  'recipe_engine/platform',
-  'recipe_engine/properties',
-  'recipe_engine/raw_io',
-  'recipe_engine/step',
-  'tryserver',
+    'git',
+    'gitiles',
+    'recipe_engine/buildbucket',
+    'recipe_engine/context',
+    'recipe_engine/file',
+    'recipe_engine/json',
+    'recipe_engine/path',
+    'recipe_engine/platform',
+    'recipe_engine/properties',
+    'recipe_engine/raw_io',
+    'recipe_engine/step',
+    'tryserver',
 ]

 from .config import config_ctx as CONFIG_CTX
--- a/recipes/recipe_modules/gclient/api.py
+++ b/recipes/recipe_modules/gclient/api.py
@ -424,3 +424,44 @@ class GclientApi(recipe_api.RecipeApi):
  @property
  def DepsDiffException(self):
    return DepsDiffException
+
+  def roll_deps(self, deps_path, dep_updates, test_data=None):
+    """Updates DEPS file to desired revisions, and returns all requried file
+    changes.
+
+    Args:
+      deps_path - Path to DEPS file that will be modified.
+      dep_updates - A map of dependencies to update (key = dependency name,
+                    value = revision).
+
+    Returns:
+      A map of all files that need to be modified (key = file path, value = file
+      content) in addition to DEPS file itself.
+      Note: that git submodules (gitlinks) are treated as files and content is a
+      commit hash.
+      Note: deps_path is not added to returned map since the repo relative path
+      is not known.
+    """
+    update_gitlink = False
+    dep_updates_args = []
+    file_changes = {}
+
+    deps_contents = self.m.file.read_text('Read DEPS file', deps_path,
+                                          test_data)
+    lines = deps_contents.split('\n')
+    for line in lines:
+      if line.startswith('git_dependencies = '):
+        if 'DEPS' not in line:
+          # Need to update gitlinks
+          update_gitlink = True
+        break
+
+    for dep, rev in dep_updates.items():
+      dep_updates_args.extend(['-r', f'{dep}@{rev}'])
+      if update_gitlink:
+        # Add gitlink updates to file changes.
+        file_changes[dep] = rev.encode('UTF-8')
+    # Apply the updates to the local DEPS files.
+    self.m.gclient('setdep',
+                   ['setdep', '--deps-file', deps_path] + dep_updates_args)
+    return file_changes
--- a/recipes/recipe_modules/gclient/examples/full.expected/basic.json
+++ b/recipes/recipe_modules/gclient/examples/full.expected/basic.json
@ -1,4 +1,42 @@
 [
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
+      "--json-output",
+      "/path/to/tmp/json",
+      "copy",
+      "foo",
+      "/path/to/tmp/"
+    ],
+    "infra_step": true,
+    "name": "Read DEPS file",
+    "~followup_annotations": [
+      "@@@STEP_LOG_LINE@foo@@@@",
+      "@@@STEP_LOG_LINE@foo@git_dependencies = 'SYNC'@@@",
+      "@@@STEP_LOG_END@foo@@@"
+    ]
+  },
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_REPO[depot_tools]/gclient.py",
+      "setdep",
+      "--deps-file",
+      "foo",
+      "-r",
+      "foo@2222222222222222222222222222222222222222"
+    ],
+    "env_suffixes": {
+      "PATH": [
+        "RECIPE_REPO[depot_tools]"
+      ]
+    },
+    "infra_step": true,
+    "name": "gclient setdep"
+  },
  {
    "cmd": [
      "vpython3",
--- a/recipes/recipe_modules/gclient/examples/full.expected/revision.json
+++ b/recipes/recipe_modules/gclient/examples/full.expected/revision.json
@ -1,4 +1,69 @@
 [
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
+      "--json-output",
+      "/path/to/tmp/json",
+      "copy",
+      "foo",
+      "/path/to/tmp/"
+    ],
+    "infra_step": true,
+    "luci_context": {
+      "realm": {
+        "name": "project:ci"
+      },
+      "resultdb": {
+        "current_invocation": {
+          "name": "invocations/build:8945511751514863184",
+          "update_token": "token"
+        },
+        "hostname": "rdbhost"
+      }
+    },
+    "name": "Read DEPS file",
+    "~followup_annotations": [
+      "@@@STEP_LOG_LINE@foo@@@@",
+      "@@@STEP_LOG_LINE@foo@git_dependencies = 'SYNC'@@@",
+      "@@@STEP_LOG_END@foo@@@"
+    ]
+  },
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_REPO[depot_tools]/gclient.py",
+      "setdep",
+      "--deps-file",
+      "foo",
+      "-r",
+      "foo@2222222222222222222222222222222222222222"
+    ],
+    "env": {
+      "DEPOT_TOOLS_REPORT_BUILD": "project/ci/builder/8945511751514863184"
+    },
+    "env_suffixes": {
+      "PATH": [
+        "RECIPE_REPO[depot_tools]"
+      ]
+    },
+    "infra_step": true,
+    "luci_context": {
+      "realm": {
+        "name": "project:ci"
+      },
+      "resultdb": {
+        "current_invocation": {
+          "name": "invocations/build:8945511751514863184",
+          "update_token": "token"
+        },
+        "hostname": "rdbhost"
+      }
+    },
+    "name": "gclient setdep"
+  },
  {
    "cmd": [
      "vpython3",
--- a/recipes/recipe_modules/gclient/examples/full.expected/tryserver.json
+++ b/recipes/recipe_modules/gclient/examples/full.expected/tryserver.json
@ -1,4 +1,69 @@
 [
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
+      "--json-output",
+      "/path/to/tmp/json",
+      "copy",
+      "foo",
+      "/path/to/tmp/"
+    ],
+    "infra_step": true,
+    "luci_context": {
+      "realm": {
+        "name": "project:try"
+      },
+      "resultdb": {
+        "current_invocation": {
+          "name": "invocations/build:8945511751514863184",
+          "update_token": "token"
+        },
+        "hostname": "rdbhost"
+      }
+    },
+    "name": "Read DEPS file",
+    "~followup_annotations": [
+      "@@@STEP_LOG_LINE@foo@@@@",
+      "@@@STEP_LOG_LINE@foo@git_dependencies = 'SYNC'@@@",
+      "@@@STEP_LOG_END@foo@@@"
+    ]
+  },
+  {
+    "cmd": [
+      "vpython3",
+      "-u",
+      "RECIPE_REPO[depot_tools]/gclient.py",
+      "setdep",
+      "--deps-file",
+      "foo",
+      "-r",
+      "foo@2222222222222222222222222222222222222222"
+    ],
+    "env": {
+      "DEPOT_TOOLS_REPORT_BUILD": "project/try/builder/8945511751514863184"
+    },
+    "env_suffixes": {
+      "PATH": [
+        "RECIPE_REPO[depot_tools]"
+      ]
+    },
+    "infra_step": true,
+    "luci_context": {
+      "realm": {
+        "name": "project:try"
+      },
+      "resultdb": {
+        "current_invocation": {
+          "name": "invocations/build:8945511751514863184",
+          "update_token": "token"
+        },
+        "hostname": "rdbhost"
+      }
+    },
+    "name": "gclient setdep"
+  },
  {
    "cmd": [
      "vpython3",
--- a/recipes/recipe_modules/gclient/examples/full.py
+++ b/recipes/recipe_modules/gclient/examples/full.py
@ -5,12 +5,14 @@
 PYTHON_VERSION_COMPATIBILITY = 'PY2+3'

 DEPS = [
-  'gclient',
-  'recipe_engine/buildbucket',
-  'recipe_engine/context',
-  'recipe_engine/path',
-  'recipe_engine/properties',
-  'recipe_engine/step',
+    'gclient',
+    'recipe_engine/assertions',
+    'recipe_engine/buildbucket',
+    'recipe_engine/context',
+    'recipe_engine/file',
+    'recipe_engine/path',
+    'recipe_engine/properties',
+    'recipe_engine/step',
 ]


@ -64,8 +66,18 @@ TEST_CONFIGS = [
    'with_tags',
 ]

+DEPS_CONTENT = """
+git_dependencies = 'SYNC'
+"""
+

 def RunSteps(api):
+  file_changes = api.gclient.roll_deps(
+      'foo', {'foo': '2222222222222222222222222222222222222222'},
+      test_data=DEPS_CONTENT)
+  api.assertions.assertEqual(
+      file_changes, {'foo': b'2222222222222222222222222222222222222222'})
+
  for config_name in TEST_CONFIGS:
    api.gclient.make_config(config_name)