- Sort repositories alphanumerically case insensitive per namespace
- Make the namespaces collapsible and store the collapsed state in local storage
Committed-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
This adds optional caches for configuration stores and
backing data files for data stores.
These stores can be enabled using the system properties
`scm.storeCache.enabled=true` and `scm.cache.dataFileCache.enabled=true`.
In addition, this adds the possibility to overwrite cache configurations
from the guice cache (see file `gcache.xml`) with system properties.
The maximum size of the external group cache for example can be
overwritten with the system property `scm.cache.externalGroups.maximumSize`.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
The new implementations are ment to be used in test
classes replacing the older implementations like
`InMemoryConfigurationStoreFactory`. The benefit
of these new classes is that the serialization using JaxB
is testet implicitly avoiding mistakes made with XML
annotations on the data classes.
Committed-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
This uses 'RetryingTest' from junit jupiter to retry integration tests that are known to fail from time to time. We explicitly mark single tests in contrast to set a global retry to be able to trace those, whenever this is intended.
To do so, we have to update to the latest version of JUnit. Unfortunately, this brought a new behaviour for the @TempDir annotation: In contrast to the former behaviour where for one test all annotated parameters got the same directory, in the new version the parameters get different directories assigned. This led to the need of some consolidation between @BeforeEach methods and the related tests.
Committed-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
- Use store name as label for repository related changes if no explicit labels are set.
- Introduce 'ignore' flag
- Fix missing call to store
- Create audit logs for permissions
- Set flex attributes for input field to use full available space
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Introduce audit log API which logs all creations,
modifications and deletions of annotated entities
and everything which is stored inside a
ConfigurationStore. Without the related Audit
Log Plugin installed this API does nothing.
- Avoiding read attempts for stores that do not exist (AbstractStore).
- Use of ReadWrite locks (everything withLockedFileForRead or withLockedFileForWrite)
- Caching of JAXB Context (TypedStoreContext.java)
- Avoid unnecessary writes to the UserGroupCache
Committed-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
In addition, this removes the links to API keys in user hal objects, when API keys are disabled.
Committed-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
Adds an overview of the permissions of a user including its groups. To do so, a new cache is introduced that stores the groups of a user, when the user is authenticated. In doing so, SCM-Manager can also list groups assigned by external authentication plugins such as LDAP. On the other hand, the user has to have been logged in at least once to get external groups, and even then the cached groups may be out of date when the overview is created. Internal groups will always be added correctly, nonetheless.
Due to the cache, another problem arised: On some logins, the xml dao for the cache failed to be read, because it was read and written at the same time. To fix this, a more thorough synchronization of the stores has been implemented.
Committed-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
To make SCM-Manager more accessible and to make it easier using scripts against the server, we created a command line interface. This command line interface can be used to perform the default actions like create, modify and delete repositories. It is also very flexible and can be extended by plugins.
The CLI already supports internationalization, help texts, input validation, loose and table-like templates and nested subcommands. Check the cli guidelines to learn how add new cli commands.
Co-authored-by: Sebastian Sdorra <sebastian.sdorra@cloudogu.com>
Enable gradle dependency locking for all subprojects. This change allows us to use dependency ranges, without loosing reproducible builds.
It also makes it easier for third party services such as snyk to detect dependencies and their versions.
Users, groups, repositories and repository roles have been sorted in the rest layer by default if no other sort option was given. In the layers "below" (aka the manager classes or the dao), the collections have been unsorted. This led to the effect, that the autocomplete resource, which did not sort all values beforehand, returned unsorted results. As a sideeffect, direct matches for an input could occur at a random position or not at all (as reported in #1695), when there were enough other matches.
With this pull request the databases for users, groups, repositories and repository roles will use instances of TreeMap instead of LinkedHashMap internally, so that these values are sorted implicitly (by id respectively name for users, groups and repository roles and namespace/name for repositories).
Due to this change the default sort applied in the rest layer could be removed.
Update Gradle to 7.3. in order to profit from the performance improvement and to prepare for dependency lockfiles. Due to the upgrade to 7.x we had to use a license plugin which supports Gradle 7.
On windows unit tests are failing because junit checks if all @tempdir directries are empty and can be deleted after test run.
But due to opened file handles (not closed resource streams) Windows keeps files, which are "in use".
Linux is less strict in this area.
Additionally I want highlight that XMLStreamReaderImpl/XMLStreamWriterImpl from apache.xerces library (in OpenJDK11 at least) which are picked at runtime as xml parser implementation - they don't close associated resources.
BTW, I thing that relying on some runtime (sometimes - unpredictable) dependencies - is bad practice, but this it up to separate topic.
Additional fix: in IOUtil is file is locked (due to permissions or opened handle) - it will undlessly try-and-retry to delete it until end of the world, on windows.
Before this change the search uses a single index which distinguishes types (repositories, users, etc.) with a field (_type).
But it has turned out that this could lead to problems, in particular if different types have the same field and uses different analyzers for those fields. The following links show even more problems of a combined index:
https://www.elastic.co/blog/index-vs-typehttps://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
With this change every type becomes its own index and the SearchEngine gets an api to modify multiple indices at once to remove all documents from all indices, which are related to a specific repository, for example.
The search uses another new api to coordinate the indexing, the central work queue.
The central work queue is able to coordinate long-running or resource intensive tasks. It is able to run tasks in parallel, but can also run tasks which targets the same resources in sequence. The queue is also persistent and can restore queued tasks after restart.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Add option to encrypt repository exports with a password and add possibility to decrypt them on repository import. Also make the repository export asynchronous. This implies that the repository export will be created on the server and can be downloaded multiple times. The repository export will be deleted automatically 10 days after creation.
The default (XML) store of SCM-Manager does not distinguish between config and config entry stores in regards to
storage locations. Nonetheless, we want to make a difference in export files, so that other store providers can handle
these stores differently. To do so, this change adds an attribute to the top level xml element of config entry stores
to mark them. In exports, these store files can now be exported in a different folder. To mark existing stores, this
introduces an update step.
* Add store exporter to collect the repository metadata
* Add EnvironmentInformationXmlGenerator
* Collect export data and put into compressed tar archive output stream
* Create full repository export endpoint.
* Add full repository export to ui
* Ignore irrelevant files from config store directory
* write metadata stores to file since a baos could teardown the server memory
* Migrate store name for git lfs files (#1504)
Changes the directory name for the git LFS blob store by
removing the repository id from the store name.
This is necessary for im- and exports of lfs blob stores,
because the original name had the repository id as a part
of it and therefore the old store would not be found when
the repository is imported with another id.
Existing blob files will be moved to the new store location
by an update step.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
* Introduce util for migrations (#1505)
With this util it is more simple to rename
or delete stores.
* Rename files in export
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
This adds a flag "archived" to repositories. Repositories marked with this can no longer be modified in any way. To do this, we switch to a new version of Shiro Static Permissions (sdorra/shiro-static-permissions#4) and specify a permission guard to check for every permission request, whether the repository in question is archived or not. Further we implement checks in stores and other activies so that no writing request may be executed by mistake.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
