From 540ba9dde07c16c0e91eea558a8b7ae15aa8c145 Mon Sep 17 00:00:00 2001 From: Takuto Ikuta Date: Wed, 19 Dec 2018 22:24:08 +0000 Subject: [PATCH] [ninjalog] add ninjalog.README.md I added document explaining ninjalog collection. This document explains what type of data is collected and how the data will be used. Bug: 900161, b/120645342 Change-Id: If47517aaa7a5510e4a634b5ccc0e4127d014dcb9 Reviewed-on: https://chromium-review.googlesource.com/c/1381912 Commit-Queue: Takuto Ikuta Reviewed-by: Dirk Pranke --- OWNERS | 3 +- ninjalog.README.md | 64 ++++++++++++++++++++++++++++++++++++ ninjalog_uploader_wrapper.py | 16 +++++---- 3 files changed, 75 insertions(+), 8 deletions(-) create mode 100644 ninjalog.README.md diff --git a/OWNERS b/OWNERS index 68a114e54..916f72804 100644 --- a/OWNERS +++ b/OWNERS @@ -17,8 +17,7 @@ per-file ninja*=scottmg@chromium.org per-file autoninja*=brucedawson@chromium.org -per-file ninjalog_uploader.py=tikuta@chromium.org -per-file ninjalog_uploader_wrapper.py=tikuta@chromium.org +per-file ninjalog*=tikuta@chromium.org per-file cbuildbot=file://CROS_OWNERS per-file chrome_set_ver=file://CROS_OWNERS diff --git a/ninjalog.README.md b/ninjalog.README.md new file mode 100644 index 000000000..b1e5284c6 --- /dev/null +++ b/ninjalog.README.md @@ -0,0 +1,64 @@ +# Ninja build log collection + +[TOC] + +## Overview + +When chromium developers use autoninja for their build, + +e.g. +``` +$ autoninja -C out/Release chrome +``` + +autoninja uploads ninja's build log for Google employees but we don't collect +logs from external contributors. + +We use [this page](https://chromium-build-stats-staging.appspot.com/should-upload) +to decide whether an autoninja user is a Googler or not. Only people with access +to Google's internal network can see 'Success'. + +Before uploading logs, autoninja shows a message 10 times to warn users that we +will collect build logs. + +autoninja users can also opt in or out by using the following commands: + +* `ninjalog_uploader_wrapper.py opt-in` +* `ninjalog_uploader_wrapper.py opt-out` + +## What type of data are collected? + +The collected build log contains + +* output file of build tasks (e.g. chrome, obj/url/url/url_util.o) +* hash of build command +* start and end time of build tasks + +See [manual of ninja](https://ninja-build.org/manual.html#ref_log) for more +details. + +In addition to ninja's build log, we send the following data for further +analysis: + +* OS (e.g. Win, Mac or Linux) +* number of cpu cores of building machine +* build targets (e.g. chrome, browser_tests) +* parallelism passed by -j flag +* following build configs + * host\_os, host\_cpu + * target\_os, target\_cpu + * symbol\_level + * use\_goma + * is\_debug + * is\_component\_build + + We don't collect personally identifiable information +(e.g. username, ip address). + +## Why ninja log is collected? / How the collected logs are used? + +We (goma team) collect build logs to find slow build tasks that harm developer's +productivity. Based on collected stats, we find the place/build tasks where we +need to focus on. Also we use collected stats to track chrome build performance +on developer's machine. We'll use this stats to measure how much we can/can't +improve build performance on developer's machine. diff --git a/ninjalog_uploader_wrapper.py b/ninjalog_uploader_wrapper.py index 722888601..9735201f3 100755 --- a/ninjalog_uploader_wrapper.py +++ b/ninjalog_uploader_wrapper.py @@ -41,8 +41,8 @@ def SaveConfig(config): def ShowMessage(countdown): print """ -Your ninjalog will be uploaded to build stats server. Uploaded log will be used -to analyze user side build performance. +Your ninjalog will be uploaded to build stats server. The uploaded log will be +used to analyze user side build performance. The following information will be uploaded with ninjalog. * OS (e.g. Win, Mac or Linux) @@ -57,13 +57,17 @@ Uploading ninjalog will be started after you run autoninja another %d time. If you don't want to upload ninjalog, please run following command. $ %s opt-out -If you allow upload ninjalog from next autoninja run, please run the following -command. +If you want to allow upload ninjalog from next autoninja run, please run the +following command. $ %s opt-in -If you have question about this, please send mail to infra-dev@chromium.org +If you have questions about this, please send mail to infra-dev@chromium.org -""" % (countdown, __file__, __file__) +You can find a more detailed explanation in +%s + +""" % (countdown, __file__, __file__, + os.path.abspath(os.path.join(THIS_DIR, "ninjalog.README.md"))) def main():