Users, groups, repositories and repository roles have been sorted in the rest layer by default if no other sort option was given. In the layers "below" (aka the manager classes or the dao), the collections have been unsorted. This led to the effect, that the autocomplete resource, which did not sort all values beforehand, returned unsorted results. As a sideeffect, direct matches for an input could occur at a random position or not at all (as reported in #1695), when there were enough other matches.
With this pull request the databases for users, groups, repositories and repository roles will use instances of TreeMap instead of LinkedHashMap internally, so that these values are sorted implicitly (by id respectively name for users, groups and repository roles and namespace/name for repositories).
Due to this change the default sort applied in the rest layer could be removed.
Update Gradle to 7.3. in order to profit from the performance improvement and to prepare for dependency lockfiles. Due to the upgrade to 7.x we had to use a license plugin which supports Gradle 7.
On windows unit tests are failing because junit checks if all @tempdir directries are empty and can be deleted after test run.
But due to opened file handles (not closed resource streams) Windows keeps files, which are "in use".
Linux is less strict in this area.
Additionally I want highlight that XMLStreamReaderImpl/XMLStreamWriterImpl from apache.xerces library (in OpenJDK11 at least) which are picked at runtime as xml parser implementation - they don't close associated resources.
BTW, I thing that relying on some runtime (sometimes - unpredictable) dependencies - is bad practice, but this it up to separate topic.
Additional fix: in IOUtil is file is locked (due to permissions or opened handle) - it will undlessly try-and-retry to delete it until end of the world, on windows.
Before this change the search uses a single index which distinguishes types (repositories, users, etc.) with a field (_type).
But it has turned out that this could lead to problems, in particular if different types have the same field and uses different analyzers for those fields. The following links show even more problems of a combined index:
https://www.elastic.co/blog/index-vs-typehttps://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
With this change every type becomes its own index and the SearchEngine gets an api to modify multiple indices at once to remove all documents from all indices, which are related to a specific repository, for example.
The search uses another new api to coordinate the indexing, the central work queue.
The central work queue is able to coordinate long-running or resource intensive tasks. It is able to run tasks in parallel, but can also run tasks which targets the same resources in sequence. The queue is also persistent and can restore queued tasks after restart.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Add option to encrypt repository exports with a password and add possibility to decrypt them on repository import. Also make the repository export asynchronous. This implies that the repository export will be created on the server and can be downloaded multiple times. The repository export will be deleted automatically 10 days after creation.
The default (XML) store of SCM-Manager does not distinguish between config and config entry stores in regards to
storage locations. Nonetheless, we want to make a difference in export files, so that other store providers can handle
these stores differently. To do so, this change adds an attribute to the top level xml element of config entry stores
to mark them. In exports, these store files can now be exported in a different folder. To mark existing stores, this
introduces an update step.
* Add store exporter to collect the repository metadata
* Add EnvironmentInformationXmlGenerator
* Collect export data and put into compressed tar archive output stream
* Create full repository export endpoint.
* Add full repository export to ui
* Ignore irrelevant files from config store directory
* write metadata stores to file since a baos could teardown the server memory
* Migrate store name for git lfs files (#1504)
Changes the directory name for the git LFS blob store by
removing the repository id from the store name.
This is necessary for im- and exports of lfs blob stores,
because the original name had the repository id as a part
of it and therefore the old store would not be found when
the repository is imported with another id.
Existing blob files will be moved to the new store location
by an update step.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
* Introduce util for migrations (#1505)
With this util it is more simple to rename
or delete stores.
* Rename files in export
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
This adds a flag "archived" to repositories. Repositories marked with this can no longer be modified in any way. To do this, we switch to a new version of Shiro Static Permissions (sdorra/shiro-static-permissions#4) and specify a permission guard to check for every permission request, whether the repository in question is archived or not. Further we implement checks in stores and other activies so that no writing request may be executed by mistake.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
There may be v1 properties with key/value pairs, where the value is not
set. Such "pairs" led to a null pointer exception in the `map` function,
because no Optional could be created for the null value.
To solve this, we filter such properties without value in the beginning,
because these properties would create a null result both ways.
* prepare license-maven-plugin for license migration
* added license mapping for tsx files and added some more excludes
* Changeover to MIT license
* Fix build problems
* Delete old remaining licenses
* Add more exclude path for license checker
* Rename included netbeans license, add exclude .m2/repository/
* Specify .m2 exclude because not only repository/, also wrapper/ must match
* Add .cache/ exclude for license check
* Modify formatting of license in java classes to comply with convention and IDE
* Add IntelliJ documentation for license configuration
* Update CHANGELOG.md
* Exclude tmp/workspace/ dir for license check
* Edit README.md
Co-authored-by: Sebastian Sdorra <sebastian.sdorra@cloudogu.com>
This is needed after migration from v1 to v2 and is done in
GitV1UpdateStep.java. Therefore we hat to make the 'forAllPaths' method
in PathBasedRepositoryLocationResolver available in the interface of
RepositoryLocationResolver.
Therefore we have to
- add an API to create stores for repository ids, not only for
repositories,
- make v1 properties available in scm-core
- make sure that properties are extracted from repositories before the
update step of a plugin runs (this is done by sorting the update steps
in a way so that "core" update steps are executed before plugin update
steps with the same version)
