buildstream.git
40 hours agoMerge pull request #1647 from jjardon/jjardon/f36 master
Jürg Billeter [Sat, 14 May 2022 12:43:21 +0000 (14:43 +0200)] 
Merge pull request #1647 from jjardon/jjardon/f36

.github: Run tests in fedora 36 also

5 days agoMerge pull request #1651 from apache/tristan/handle-asset-cache-error
Tristan Van Berkom [Tue, 10 May 2022 22:51:33 +0000 (07:51 +0900)] 
Merge pull request #1651 from apache/tristan/handle-asset-cache-error

_elementsourcescache.py: Properly handle AssetCacheError

5 days ago_elementsourcescache.py: Properly handle AssetCacheError tristan/handle-asset-cache-error 1651/head
Tristan van Berkom [Tue, 10 May 2022 22:49:39 +0000 (07:49 +0900)] 
_elementsourcescache.py: Properly handle AssetCacheError

5 days agoMerge pull request #1589 from ssssam/sam/badge-cleanup
Tristan Van Berkom [Tue, 10 May 2022 10:40:10 +0000 (19:40 +0900)] 
Merge pull request #1589 from ssssam/sam/badge-cleanup

README.rst: Cleanup shields

5 days agoMerge branch 'master' into sam/badge-cleanup 1589/head
Tristan Van Berkom [Tue, 10 May 2022 10:37:34 +0000 (19:37 +0900)] 
Merge branch 'master' into sam/badge-cleanup

5 days agoMerge pull request #1648 from nanonyme/new-git
Tristan Van Berkom [Tue, 10 May 2022 10:26:57 +0000 (19:26 +0900)] 
Merge pull request #1648 from nanonyme/new-git

Ensure the initial branch in git is master

5 days agoMerge pull request #1646 from apache/tristan/sourcecache-retries
Tristan Van Berkom [Tue, 10 May 2022 10:10:01 +0000 (19:10 +0900)] 
Merge pull request #1646 from apache/tristan/sourcecache-retries

Retry attempts to pull and push sources to remote source caches

8 days agoEnsure the initial branch in git is master 1648/head
Seppo Yli-Olli [Sat, 7 May 2022 19:36:55 +0000 (22:36 +0300)] 
Ensure the initial branch in git is master

There's many tests depending on this all around and default
branch is changing to main so it's better to be explicit.

8 days ago.github: Run tests in fedora 36 1647/head
Javier Jardón [Sat, 7 May 2022 17:15:15 +0000 (18:15 +0100)] 
.github: Run tests in fedora 36

And remove soon to be deprecated fedora 34

8 days ago_sourcecache.py: Refactor to handle AssetCacheError properly 1646/head
Tristan van Berkom [Sat, 7 May 2022 09:50:06 +0000 (18:50 +0900)] 
_sourcecache.py: Refactor to handle AssetCacheError properly

This file appears to have fallen out of sync and had incorrect assumptions
about the errors which can be raised from AssetRemote.fetch_directory()
and AssetCache.push_directory().

This patch also concentrates the `try` blocks around the functions which
can raise AssetCacheError.

8 days agoMake source cache errors temporary
Seppo Yli-Olli [Fri, 6 May 2022 19:27:31 +0000 (22:27 +0300)] 
Make source cache errors temporary

4 weeks agoMerge pull request #1607 from apache/tristan/split-source-methods
Tristan Van Berkom [Fri, 15 Apr 2022 09:56:45 +0000 (18:56 +0900)] 
Merge pull request #1607 from apache/tristan/split-source-methods

source.py: Split up staging functions into separate codepaths.

4 weeks agosource.py: Split up staging functions into separate codepaths. tristan/split-source-methods 1607/head
Tristan van Berkom [Thu, 3 Mar 2022 08:57:38 +0000 (17:57 +0900)] 
source.py: Split up staging functions into separate codepaths.

Currently we pass a file path string or Directory object to these methods
depending on whether BST_STAGE_VIRTUAL_DIRECTORY is set, which is not very
pretty for plugin developers especially if they are using type annotations
and need to handle `Union[str, Directory]` in these methods.

Instead, created Source.stage_directory() and Source.init_workspace_directory()
to handle the Directory variants of these methods separately.

Also updated `local` and `workspace` plugins to use the new methods.

4 weeks agoMerge pull request #1640 from apache/tristan/extend-cachekey-tests
Tristan Van Berkom [Fri, 15 Apr 2022 09:14:17 +0000 (18:14 +0900)] 
Merge pull request #1640 from apache/tristan/extend-cachekey-tests

tests/cachekey: Adding test for default project.conf variables

4 weeks agotests/cachekey: Adding test for default project.conf variables tristan/extend-cachekey-tests 1640/head
Tristan van Berkom [Fri, 15 Apr 2022 08:07:50 +0000 (17:07 +0900)] 
tests/cachekey: Adding test for default project.conf variables

4 weeks agoMerge pull request #1639 from apache/tristan/add-tox-release
Tristan Van Berkom [Wed, 13 Apr 2022 07:56:48 +0000 (16:56 +0900)] 
Merge pull request #1639 from apache/tristan/add-tox-release

tox.ini: Add convenience hook for publishing releases

4 weeks agotox.ini: Add convenience hook for publishing releases tristan/add-tox-release 1639/head
Tristan van Berkom [Wed, 13 Apr 2022 07:55:11 +0000 (16:55 +0900)] 
tox.ini: Add convenience hook for publishing releases

Because I always forget how to do this and end up asking Chandan, now
I should be able to remember how to do this.

4 weeks agoMerge pull request #1638 from apache/tristan/update-porting-guide
Tristan Van Berkom [Wed, 13 Apr 2022 07:48:20 +0000 (16:48 +0900)] 
Merge pull request #1638 from apache/tristan/update-porting-guide

Update porting guide

4 weeks agodoc/source/porting_project.rst: Updating for plugins move tristan/update-porting-guide 1638/head
Tristan van Berkom [Wed, 13 Apr 2022 07:33:56 +0000 (16:33 +0900)] 
doc/source/porting_project.rst: Updating for plugins move

  o Update table regarding where plugins have moved
  o Update example to refer to buildstream-plugins

4 weeks agoMerge pull request #1636 from apache/tristan/remove-plugins
Tristan Van Berkom [Wed, 13 Apr 2022 05:14:25 +0000 (14:14 +0900)] 
Merge pull request #1636 from apache/tristan/remove-plugins

Remove plugins

5 weeks agotox.ini: Use version of bst-plugins-experimental which doesn't try to test upstream git tristan/remove-plugins 1636/head
Tristan van Berkom [Wed, 6 Apr 2022 10:19:45 +0000 (19:19 +0900)] 
tox.ini: Use version of bst-plugins-experimental which doesn't try to test upstream git

As described here: https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/175

5 weeks agosrc/buildstream/plugins: Removing autotools element
Tristan van Berkom [Wed, 6 Apr 2022 09:08:45 +0000 (18:08 +0900)] 
src/buildstream/plugins: Removing autotools element

Also making some ammendments to the documentation affected by removing
the autotools element.

5 weeks agotox.ini, doc/examples: Use autotools element from buildstream-plugins
Tristan van Berkom [Wed, 6 Apr 2022 08:53:40 +0000 (17:53 +0900)] 
tox.ini, doc/examples: Use autotools element from buildstream-plugins

5 weeks agotests/remoteexecution: Use internal autotools element for remote execution tests
Tristan van Berkom [Wed, 6 Apr 2022 08:35:43 +0000 (17:35 +0900)] 
tests/remoteexecution: Use internal autotools element for remote execution tests

5 weeks agotests/remotecache/simple.py: Use internal autotools element for the build
Tristan van Berkom [Wed, 6 Apr 2022 08:27:23 +0000 (17:27 +0900)] 
tests/remotecache/simple.py: Use internal autotools element for the build

5 weeks agotests/integration: Use internal autotools element for autotools builds
Tristan van Berkom [Wed, 6 Apr 2022 08:17:04 +0000 (17:17 +0900)] 
tests/integration: Use internal autotools element for autotools builds

5 weeks agotests/format/projectoverrides.py: Use internal autotools plugin to test the overrides
Tristan van Berkom [Wed, 6 Apr 2022 08:02:29 +0000 (17:02 +0900)] 
tests/format/projectoverrides.py: Use internal autotools plugin to test the overrides

5 weeks agotests/format/variables.py: Use internal autotools plugin to test variable compositing
Tristan van Berkom [Wed, 6 Apr 2022 07:56:25 +0000 (16:56 +0900)] 
tests/format/variables.py: Use internal autotools plugin to test variable compositing

5 weeks agotests/plugins/sample-plugins: Adding autotools plugin for local use
Tristan van Berkom [Wed, 6 Apr 2022 07:34:37 +0000 (16:34 +0900)] 
tests/plugins/sample-plugins: Adding autotools plugin for local use

5 weeks agotests: Using manual elements instead of autotools elements where possible
Tristan van Berkom [Tue, 5 Apr 2022 08:10:12 +0000 (17:10 +0900)] 
tests: Using manual elements instead of autotools elements where possible

Affected tests:

  - _testing/_sourcetests/project (standard source tests)
  - tests/frontend/exceptions
  - tests/frontend/completions.py
  - tests/frontend/project/elements/multiple_targets
  - tests/format/dependencies.py
  - tests/format/assertion.py
  - tests/format/option-overrides
  - tests/format/optionarch.py
  - tests/format/optionbool.py
  - tests/format/optioneltmask.py
  - tests/format/optionflags.py
  - tests/format/optionos.py
  - tests/format/optionexports.py
  - tests/format/optionenum.py
  - tests/format/options.py
  - tests/format/listdirectiveerrors.py
  - tests/format/project.py
  - tests/sourcecache/config.py
  - tests/artifactcache/config.py
  - tests/sources/tar.py
  - tests/sources/remote.py
  - tests/sandboxes/remote-exec-config.py

5 weeks agoplugins/sources/git.py: Removing git source
Tristan van Berkom [Wed, 30 Mar 2022 05:39:38 +0000 (14:39 +0900)] 
plugins/sources/git.py: Removing git source

And removing the git source from the documentation plugins index

5 weeks agotests/cachekey: Removing git source from cache key tests
Tristan van Berkom [Wed, 30 Mar 2022 05:38:32 +0000 (14:38 +0900)] 
tests/cachekey: Removing git source from cache key tests

5 weeks agotests/conftest.py: Remove git from generic source tests
Tristan van Berkom [Wed, 30 Mar 2022 05:36:30 +0000 (14:36 +0900)] 
tests/conftest.py: Remove git from generic source tests

5 weeks agotests/sources/git.py: Remove git source tests
Tristan van Berkom [Wed, 30 Mar 2022 05:35:36 +0000 (14:35 +0900)] 
tests/sources/git.py: Remove git source tests

5 weeks agotests/frontend/mirror.py: Use internal git source for these tests
Tristan van Berkom [Wed, 30 Mar 2022 03:55:49 +0000 (12:55 +0900)] 
tests/frontend/mirror.py: Use internal git source for these tests

5 weeks agotests/frontend/track.py: port tests to use internal git source
Tristan van Berkom [Tue, 29 Mar 2022 07:41:35 +0000 (16:41 +0900)] 
tests/frontend/track.py: port tests to use internal git source

In order to remove the `git` plugin, we're porting some of our test cases
which we need to keep to use the private `git` plugin added to the test
sample-plugins package.

5 weeks agotests/plugins/sample-plugins: Adding git source to sample plugins
Tristan van Berkom [Tue, 29 Mar 2022 06:49:07 +0000 (15:49 +0900)] 
tests/plugins/sample-plugins: Adding git source to sample plugins

5 weeks agotests/testutils/site.py: Move the pip_sample_plugins() check to a common site.py
Tristan van Berkom [Tue, 29 Mar 2022 06:45:27 +0000 (15:45 +0900)] 
tests/testutils/site.py: Move the pip_sample_plugins() check to a common site.py

As we will be using the sample plugins package to carry some plugins we want
to use for general purpose testing of the API surface throughout buildstream,
we'll need to have this in a shared location.

5 weeks agotests: Use tar for testing in place of git wherever possible
Tristan van Berkom [Mon, 28 Mar 2022 10:10:51 +0000 (19:10 +0900)] 
tests: Use tar for testing in place of git wherever possible

Affected tests and files:

  - tests/format/project.py
  - tests/format/include.py
  - tests/frontend/cross_junction_workspace.py
  - tests/frontend/logging.py
  - tests/frontend/order.py
  - tests/frontend/workspace.py
  - tests/frontend/default_target.py
  - tests/frontend/buildcheckout.py
  - tests/frontend/pull.py
  - tests/sourcecache/push.py
  - tests/sourcecache/fetch.py
  - tests/sources/no_fetch_cached.py
  - tests/elements/filter.py
  - tests/remoteexecution/junction.py
  - tests/testutils/junction.py: Port generate_junction() to use tar repo instead of git repo
  - tests/testutils/element_generators.py: Port create_element_size() to use tar repo
  - _testing/_utils/junction.py: Use tar for generating junctions in source tests

5 weeks agoplugins/sources/patch.py: Removing patch source
Tristan van Berkom [Mon, 28 Mar 2022 08:03:46 +0000 (17:03 +0900)] 
plugins/sources/patch.py: Removing patch source

And removing patch source from documentation plugins index.

5 weeks agotests/sourcecache/project: Adding patch source to the project so tests pass
Tristan van Berkom [Mon, 28 Mar 2022 08:47:33 +0000 (17:47 +0900)] 
tests/sourcecache/project: Adding patch source to the project so tests pass

5 weeks agotests/cachekey: Removing patch source from cache key tests
Tristan van Berkom [Mon, 28 Mar 2022 08:02:13 +0000 (17:02 +0900)] 
tests/cachekey: Removing patch source from cache key tests

5 weeks agotests/sources/patch.py: Removing patch source tests
Tristan van Berkom [Mon, 28 Mar 2022 07:59:03 +0000 (16:59 +0900)] 
tests/sources/patch.py: Removing patch source tests

5 weeks agoplugins/sources/pip.py: Removing pip source
Tristan van Berkom [Mon, 28 Mar 2022 07:39:51 +0000 (16:39 +0900)] 
plugins/sources/pip.py: Removing pip source

And removing pip from the docs plugins index.

5 weeks agotests/cachekey: Removing pip source from cachekey test
Tristan van Berkom [Mon, 28 Mar 2022 07:38:42 +0000 (16:38 +0900)] 
tests/cachekey: Removing pip source from cachekey test

5 weeks agotests/integration/pip_source.py: Removing pip source integration test
Tristan van Berkom [Mon, 28 Mar 2022 07:33:26 +0000 (16:33 +0900)] 
tests/integration/pip_source.py: Removing pip source integration test

5 weeks agotests/sources/pip.py: Removing pip source test
Tristan van Berkom [Mon, 28 Mar 2022 07:32:11 +0000 (16:32 +0900)] 
tests/sources/pip.py: Removing pip source test

5 weeks agoplugins/sources/bzr.py: Removing bzr source
Tristan van Berkom [Mon, 28 Mar 2022 07:13:26 +0000 (16:13 +0900)] 
plugins/sources/bzr.py: Removing bzr source

And removing bzr from the docs plugins index.

5 weeks agotests/cachekey: Removing bzr from cachekey tests
Tristan van Berkom [Mon, 28 Mar 2022 07:12:23 +0000 (16:12 +0900)] 
tests/cachekey: Removing bzr from cachekey tests

5 weeks agotests/sources/bzr.py: Removing bzr specific tests
Tristan van Berkom [Mon, 28 Mar 2022 07:09:21 +0000 (16:09 +0900)] 
tests/sources/bzr.py: Removing bzr specific tests

5 weeks agotests/conftest.py: Remove bzr from automated source tests
Tristan van Berkom [Mon, 28 Mar 2022 07:08:25 +0000 (16:08 +0900)] 
tests/conftest.py: Remove bzr from automated source tests

5 weeks ago_testing/_cachekeys.py: Fix references to buildstream._testing
Tristan van Berkom [Wed, 30 Mar 2022 03:13:20 +0000 (12:13 +0900)] 
_testing/_cachekeys.py: Fix references to buildstream._testing

5 weeks agodoc/source/porting_project.rst: Correcting example of junction plugin in porting...
Tristan van Berkom [Tue, 5 Apr 2022 08:48:35 +0000 (17:48 +0900)] 
doc/source/porting_project.rst: Correcting example of junction plugin in porting guide

5 weeks ago.asf.yaml: Whitespace commit, trying to trigger docs builds
Tristan van Berkom [Wed, 6 Apr 2022 11:59:31 +0000 (20:59 +0900)] 
.asf.yaml: Whitespace commit, trying to trigger docs builds

5 weeks agoMerge pull request #1637 from apache/tristan/use-latest-bst-plugins-experimental
Tristan Van Berkom [Tue, 5 Apr 2022 08:26:12 +0000 (17:26 +0900)] 
Merge pull request #1637 from apache/tristan/use-latest-bst-plugins-experimental

tox.ini: Test against latest bst-plugins-experimental tag

5 weeks agotox.ini: Test against latest bst-plugins-experimental tag tristan/use-latest-bst-plugins-experimental 1637/head
Tristan van Berkom [Tue, 5 Apr 2022 07:28:49 +0000 (16:28 +0900)] 
tox.ini: Test against latest bst-plugins-experimental tag

This has the fix to work with zip source removed from buildstream

6 weeks agoMerge pull request #1635 from apache/tristan/fix-merge-ci
Tristan Van Berkom [Wed, 30 Mar 2022 03:07:26 +0000 (12:07 +0900)] 
Merge pull request #1635 from apache/tristan/fix-merge-ci

.github/workflows: Fix docs builds in merge

6 weeks ago.github/workflows: Fix docs builds in merge tristan/fix-merge-ci 1635/head
Tristan van Berkom [Wed, 30 Mar 2022 03:05:49 +0000 (12:05 +0900)] 
.github/workflows: Fix docs builds in merge

Complete the work done in #1634, run the other workflows on ubuntu 18.04

6 weeks agoMerge pull request #1634 from apache/abderrahim/test-ubuntu
Tristan Van Berkom [Tue, 29 Mar 2022 11:28:40 +0000 (20:28 +0900)] 
Merge pull request #1634 from apache/abderrahim/test-ubuntu

.github/workflow: run tests using ubuntu 18.04

6 weeks ago.github/workflow: run tests using ubuntu 18.04 abderrahim/test-ubuntu 1634/head
Abderrahim Kitouni [Mon, 28 Mar 2022 08:01:43 +0000 (09:01 +0100)] 
.github/workflow: run tests using ubuntu 18.04

Ubuntu 20.04 seems to have a kernel bug making tests that use buildbox-fuse
hang.

6 weeks agoMerge pull request #1633 from apache/tristan/update-black
Tristan Van Berkom [Tue, 29 Mar 2022 08:27:20 +0000 (17:27 +0900)] 
Merge pull request #1633 from apache/tristan/update-black

tox.ini: Updating black to version 22.3.0

6 weeks agoAll: Updating reformatted sources with new black version. tristan/update-black 1633/head
Tristan van Berkom [Tue, 29 Mar 2022 07:54:04 +0000 (16:54 +0900)] 
All: Updating reformatted sources with new black version.

6 weeks agotox.ini: Updating black to version 22.3.0
Tristan van Berkom [Tue, 29 Mar 2022 07:51:40 +0000 (16:51 +0900)] 
tox.ini: Updating black to version 22.3.0

This is required to avoid a recently occurring exception when running black,
as outlined here: https://github.com/psf/black/issues/2964

6 weeks agoMerge pull request #1630 from apache/tristan/remove-sip-zource
Tristan Van Berkom [Mon, 28 Mar 2022 06:25:54 +0000 (15:25 +0900)] 
Merge pull request #1630 from apache/tristan/remove-sip-zource

Remove zip source

7 weeks agodoc/source/core_plugins.rst: Remove zip source from docs index tristan/remove-sip-zource 1630/head
Tristan van Berkom [Sat, 26 Mar 2022 08:46:38 +0000 (17:46 +0900)] 
doc/source/core_plugins.rst: Remove zip source from docs index

7 weeks agoplugins/sources/zip.py: Removing zip source
Tristan van Berkom [Sat, 26 Mar 2022 06:18:45 +0000 (15:18 +0900)] 
plugins/sources/zip.py: Removing zip source

7 weeks agotests/cachekey: Removing zip sources from the cache key tests
Tristan van Berkom [Sat, 26 Mar 2022 06:39:26 +0000 (15:39 +0900)] 
tests/cachekey: Removing zip sources from the cache key tests

7 weeks agotests/sources/zip.py: Removing zip tests
Tristan van Berkom [Sat, 26 Mar 2022 06:17:56 +0000 (15:17 +0900)] 
tests/sources/zip.py: Removing zip tests

7 weeks agotests/testutils/repo/zip.py: Removing Zip test repo scaffold
Tristan van Berkom [Sat, 26 Mar 2022 06:14:33 +0000 (15:14 +0900)] 
tests/testutils/repo/zip.py: Removing Zip test repo scaffold

7 weeks agoMerge pull request #1629 from apache/tristan/remove-old-badges
Tristan Van Berkom [Sat, 26 Mar 2022 06:03:30 +0000 (15:03 +0900)] 
Merge pull request #1629 from apache/tristan/remove-old-badges

README.rst: Remove old gitlab badges

7 weeks agoREADME.rst: Remove old gitlab badges tristan/remove-old-badges 1629/head
Tristan van Berkom [Fri, 25 Mar 2022 05:28:43 +0000 (14:28 +0900)] 
README.rst: Remove old gitlab badges

We still had some badges which refer to the old gitlab project site.

8 weeks agoMerge pull request #1546 from abderrahim/yaml-deprecation
Abderrahim Kitouni [Sun, 20 Mar 2022 15:21:47 +0000 (16:21 +0100)] 
Merge pull request #1546 from abderrahim/yaml-deprecation

_yaml.pyx: port roundtripping code to the new ruamel.yaml API

8 weeks agoMerge pull request #1618 from apache/abderrahim/fix-tests
Abderrahim Kitouni [Sun, 20 Mar 2022 15:20:59 +0000 (16:20 +0100)] 
Merge pull request #1618 from apache/abderrahim/fix-tests

.github/run-ci.sh: remove extra quotes

8 weeks agoMerge pull request #1592 from abderrahim/reqs
Abderrahim Kitouni [Sun, 20 Mar 2022 09:52:00 +0000 (10:52 +0100)] 
Merge pull request #1592 from abderrahim/reqs

update requirements (and fix new linter errors)

8 weeks ago_yaml.pyx: port roundtripping code to the new ruamel.yaml API 1546/head
Abderrahim Kitouni [Mon, 13 Dec 2021 13:37:53 +0000 (14:37 +0100)] 
_yaml.pyx: port roundtripping code to the new ruamel.yaml API

Also drop roundtrip_load_data that was unused

Fixes #1495

8 weeks agoupdate requirements (and fix new linter errors) 1592/head
Abderrahim Kitouni [Tue, 15 Feb 2022 11:29:58 +0000 (12:29 +0100)] 
update requirements (and fix new linter errors)

8 weeks agoMerge pull request #1626 from apache/tristan/add-missing-symbol
Tristan Van Berkom [Fri, 18 Mar 2022 08:55:06 +0000 (17:55 +0900)] 
Merge pull request #1626 from apache/tristan/add-missing-symbol

__init__.py: Adding FastEnum

8 weeks ago__init__.py: Adding FastEnum tristan/add-missing-symbol 1626/head
Tristan van Berkom [Fri, 18 Mar 2022 08:41:41 +0000 (17:41 +0900)] 
__init__.py: Adding FastEnum

The BuildStream API is constructed such that plugins can import any
data type from the main package without digging into subpackages, it
was missing FastEnum.

8 weeks agoMerge pull request #1427 from apache/bschubert/remove-multiline-switch-for-re
Tristan Van Berkom [Fri, 18 Mar 2022 08:33:16 +0000 (17:33 +0900)] 
Merge pull request #1427 from apache/bschubert/remove-multiline-switch-for-re

utils.py: Remove the (?ms) switch at the start of _glob2re

8 weeks agoutils.py: Replace the (?ms) switch by equivalent re. options bschubert/remove-multiline-switch-for-re 1427/head
Benjamin Schubert [Fri, 28 Aug 2020 13:57:24 +0000 (13:57 +0000)] 
utils.py: Replace the (?ms) switch by equivalent re. options

The 'm' switch stands for multiline output.
The 's' makes '.' also match newlines.

It is problematic in mutliple places where its use is deprecated,
as now they need to be at the start of an expression, which is not the
case in all the calls to it that we are doing.

If someone wants the same functionality, they should add it explictely
after calling _glob2re

8 weeks agoMerge pull request #1624 from apache/tristan/add-abderrahim-to-codeowners
Tristan Van Berkom [Fri, 18 Mar 2022 07:00:11 +0000 (16:00 +0900)] 
Merge pull request #1624 from apache/tristan/add-abderrahim-to-codeowners

.github/CODEOWNERS: Adding Abderrahim to the CODEOWNERS

8 weeks ago.github/CODEOWNERS: Adding Abderrahim to the CODEOWNERS tristan/add-abderrahim-to-codeowners 1624/head
Tristan van Berkom [Fri, 18 Mar 2022 06:59:30 +0000 (15:59 +0900)] 
.github/CODEOWNERS: Adding Abderrahim to the CODEOWNERS

8 weeks ago.github/run-ci.sh: remove extra quotes abderrahim/fix-tests 1618/head
Abderrahim Kitouni [Thu, 17 Mar 2022 07:39:01 +0000 (08:39 +0100)] 
.github/run-ci.sh: remove extra quotes

The quotes prevent passing multiple tests (or no test) on the command line

Fixes ec84e3f113e991c6824d3fa16c105cf2be0e7f50

8 weeks agoMerge pull request #1622 from apache/tristan/fix-plugins-ci
Tristan Van Berkom [Fri, 18 Mar 2022 06:32:27 +0000 (15:32 +0900)] 
Merge pull request #1622 from apache/tristan/fix-plugins-ci

tox.ini: Update bst-plugins-external ref to pass CI

8 weeks agotox.ini: Update bst-plugins-external ref to pass CI tristan/fix-plugins-ci 1622/head
Tristan van Berkom [Fri, 18 Mar 2022 06:30:42 +0000 (15:30 +0900)] 
tox.ini: Update bst-plugins-external ref to pass CI

The plugins CI which tests bst-plugins-experimental against current buildstream
needed to be updated to use a reference to bst-plugins-experimental which has
been updated to use newly changed buildstream APIs.

8 weeks agoMerge pull request #1617 from apache/tristan/sandbox-run-cleanup
Tristan Van Berkom [Fri, 18 Mar 2022 06:13:27 +0000 (15:13 +0900)] 
Merge pull request #1617 from apache/tristan/sandbox-run-cleanup

Cleanup `Sandbox.run()` batching API

8 weeks agosandbox.py: Remove `artifact` parameter from `Sandbox.mark_directory()` tristan/sandbox-run-cleanup 1617/head
Tristan van Berkom [Wed, 16 Mar 2022 05:56:16 +0000 (14:56 +0900)] 
sandbox.py: Remove `artifact` parameter from `Sandbox.mark_directory()`

This is no longer observed, and the API is clearer this way.

Sandbox.mark_directory() now simply:
  o Ensures a directory will exist in the sandbox
  o Ensures that directory will be read-write within the sandbox, even when
    the root directory is read-only

8 weeks agotests/frontend/buildcheckout.py: Remove obsolete test case
Tristan van Berkom [Wed, 16 Mar 2022 05:18:20 +0000 (14:18 +0900)] 
tests/frontend/buildcheckout.py: Remove obsolete test case

This test dates back to 2017 in commit 10847a66dc0748696829d5fd18ad17043c2c6878
and checks for there to be an error when depending on an artifact which had
installed files where the build directory is presumed to be.

Upon investigation, the test detects the expected error but for the wrong
reason, there is no conflict when staging files to the build directory but
instead an error is reported because there is no /install-root in the sandbox
when the manual build element completes it's build.

Remove this.

8 weeks agobuildelement.py: Removing obsolete warning
Tristan van Berkom [Wed, 16 Mar 2022 05:15:31 +0000 (14:15 +0900)] 
buildelement.py: Removing obsolete warning

This warning originates in 2017 from commit 10847a66dc0748696829d5fd18ad17043c2c6878

This is obsolete because:

  o The logic is incorrect, build directories by default are now namespaced
    by their element name, so the warning no longer makes sense.

  o The check currently checks for "/buildstream/build" on the host
    system rather than in the sandbox in order to perform the aforementioned
    invalid check.

Remove this.

8 weeks agoelement.py, buildelement.py: Remove prepare() method and related mechanics
Tristan van Berkom [Wed, 16 Mar 2022 05:03:44 +0000 (14:03 +0900)] 
element.py, buildelement.py: Remove prepare() method and related mechanics

This removes the distinction of prepare()/assemble() which is obsolete since
refactoring workspaced incremental builds to work in a remote execution context.

Fixes #1293

8 weeks agoMerge pull request #1620 from apache/tristan/fix-cas-cache-tests
Tristan Van Berkom [Fri, 18 Mar 2022 05:43:29 +0000 (14:43 +0900)] 
Merge pull request #1620 from apache/tristan/fix-cas-cache-tests

tests/testutils/artifactshare.py: Use 127.0.0.1 instead of localhost for CAS server tests

8 weeks agotests/testutils/artifactshare.py: Use 127.0.0.1 instead of localhost for CAS server... tristan/fix-cas-cache-tests 1620/head
Tristan van Berkom [Fri, 18 Mar 2022 04:44:46 +0000 (13:44 +0900)] 
tests/testutils/artifactshare.py: Use 127.0.0.1 instead of localhost for CAS server tests

In some CI runners this is needed, otherwise the tests which spin up a CAS
server hang for a while before timing out trying to open a port on "localhost:0".

8 weeks agoMerge pull request #1609 from apache/tristan/virtual-directory-cleanup
Tristan Van Berkom [Thu, 17 Mar 2022 06:13:08 +0000 (15:13 +0900)] 
Merge pull request #1609 from apache/tristan/virtual-directory-cleanup

Directory API refactoring

8 weeks agostorage/directory.py: Rename descend() -> open_directory() tristan/virtual-directory-cleanup 1609/head
Tristan van Berkom [Wed, 16 Mar 2022 09:55:45 +0000 (18:55 +0900)] 
storage/directory.py: Rename descend() -> open_directory()

Since we support ".." entries in this API this appears to be a better
name for the API and is consistent with the existing open_file() API name.

8 weeks agostorage/directory.py: Remove is_empty() in favor of implementing __len__()
Tristan van Berkom [Wed, 16 Mar 2022 07:24:01 +0000 (16:24 +0900)] 
storage/directory.py: Remove is_empty() in favor of implementing __len__()

This is a more pythonic approach for the API.

8 weeks agostorage/directory.py: Make get_size() an internal method
Tristan van Berkom [Sun, 13 Mar 2022 05:37:32 +0000 (14:37 +0900)] 
storage/directory.py: Make get_size() an internal method

This is not needed by plugin authors and is only used internally to
calculate the size of an artifact being cached.

8 weeks agostorage/directory.py: Switch API to use path strings instead of lists
Tristan van Berkom [Sat, 12 Mar 2022 05:08:03 +0000 (14:08 +0900)] 
storage/directory.py: Switch API to use path strings instead of lists

This is a more natural and intuitive API, documented how path strings
work with "/" separators and support for "." and ".." elements.

2 months agostorage/directory.py: Revisit FileStat definition, revive FileType
Tristan van Berkom [Fri, 11 Mar 2022 04:41:36 +0000 (13:41 +0900)] 
storage/directory.py: Revisit FileStat definition, revive FileType

After discussing this with Jürg we decided to make the file type an
enumeration again.

2 months agostorage/directory.py: Make Directory.export_files() an internal function
Tristan van Berkom [Thu, 10 Mar 2022 05:26:18 +0000 (14:26 +0900)] 
storage/directory.py: Make Directory.export_files() an internal function

This is not needed by plugin authors.

2 months agostorage/directory.py: Remove `can_link` parameter from Directory.import_files()
Tristan van Berkom [Thu, 10 Mar 2022 05:03:02 +0000 (14:03 +0900)] 
storage/directory.py: Remove `can_link` parameter from Directory.import_files()

This is a historic artifact from when we used to hardlink files into a staging
area which is mostly rendered obsolete by allowing buildbox to handle these
details, it is ignored by the CasBasedDirectory completely, and we don't use
it anywhere with the FileBasedDirectory.

Remove the option entirely from public and private import_files() codepaths.