Skip to main content

Check out Port for yourself ➜ 

Capabilities

This page describes the data ingestion capabilities of Port's GitLab integration.

Enrich entities with file contents​

You can enrich project, folder, and file entities with the contents of specific files from your repositories. This is useful for including README files, CODEOWNERS, or other documentation directly in your entity properties.

To do this, use the includedFiles selector to specify which files to fetch, and then reference them in your mapping using .__includedFiles["<file_path>"].

resources:
  - kind: project
    selector:
      query: 'true'
      includedFiles:
        - README.md
        - CODEOWNERS
    port:
      entity:
        mappings:
          identifier: .path_with_namespace | gsub(" "; "")
          title: .name
          blueprint: '"service"'
          properties:
            readme: .__includedFiles["README.md"]
            codeowners: .__includedFiles["CODEOWNERS"]
            url: .web_url
Deprecation notice

The file:// prefix in mappings is deprecated and will be removed in a future version. Use the includedFiles selector instead. The file:// prefix will continue to work but will show deprecation warnings in the logs.

The includedFiles feature:

  • Fetches file contents from the project's default branch (or specified branch for folder/file entities).
  • Stores file contents in the __includedFiles object in the raw data.
  • Handles missing files gracefully (stores null if a file doesn't exist).
  • Works with project, folder, and file entity types.

Enrich entities with search queries​

You can enrich project entities with search query results to check for the existence of files, patterns, or content in your repositories. This is useful for compliance checks, policy enforcement, and scorecard evaluations.

To do this, use the searchQueries selector to specify which search queries to execute, and then reference the results in your mapping using .__searchQueries["<name>"].

resources:
  - kind: project
    selector:
      query: 'true'
      searchQueries:
        - name: hasPortConfig
          scope: blobs
          query: 'filename:port.yml'
        - name: hasCI
          scope: blobs
          query: 'filename:.gitlab-ci.yml'
        - name: hasDockerfile
          scope: blobs
          query: 'filename:Dockerfile'
    port:
      entity:
        mappings:
          identifier: .path_with_namespace | gsub(" "; "")
          title: .name
          blueprint: '"service"'
          properties:
            hasPortConfig: .__searchQueries["hasPortConfig"]
            hasCI: .__searchQueries["hasCI"]
            hasDockerfile: .__searchQueries["hasDockerfile"]
            url: .web_url

The searchQueries feature:

  • Executes search queries using the GitLab Advanced Search API.
  • Stores results in the __searchQueries object in the raw data as boolean values (true if matches found, false otherwise).
  • Supports multiple search scopes (e.g., blobs, commits, wiki_blobs).
  • Handles errors gracefully (stores null if a query fails).
  • Works with project entities.

Each search query requires:

  • name - A unique name for this search query, used as the key in __searchQueries.
  • scope - The GitLab search scope (defaults to blobs if not specified).
  • query - The search query string following GitLab Advanced Search syntax.
Deprecation notice

The search:// prefix in mappings is deprecated and will be removed in a future version. Use the searchQueries selector instead. The search:// prefix will continue to work but will show deprecation warnings in the logs.

Ingest files from your repositories​

Port allows you to fetch JSON and YAML files from your repositories, and create entities from them in your software catalog. This is done using the file kind in your GitLab mapping configuration.

For example, say you want to manage your package.json files in Port. You will need to create a manifest blueprint, with each of its entities representing a package.json file.

The following configuration fetches all package.json files from my-project and my-other-project, and creates an entity for each of them, based on the manifest blueprint:

resources:
  - kind: file
    selector:
      query: 'true'
      files:
        path: 'package.json'
        repos:
          # Optional: omit to search across all top-most level groups you have access to
          # Note: repositories not part of any group are excluded from the search
          # Replace with your repository's path_with_namespace (e.g., "group/project" or "group/subgroup/project")
          - group/my-project
          - group/my-other-project

The file kind follows GitLab's Advanced Search type, adhering to its syntax, limitations, and capabilities.

Path field capabilities​

The path parameter uses GitLab's Advanced Search syntax, which differs from glob patterns. The search supports:

  • Simple * wildcards for filename patterns: *.tf, package.json, test_*.
  • Exact file paths: src/app/main.py, infra/terraform/main.tf.

The search doesn't support:

  • Path wildcards like infra/terraform/*.tf.
  • Recursive patterns like **/filename.
  • Complex patterns like *.{js,ts}, [abc]*, and !exclude.

For complex patterns, you can split them into multiple file kinds or specify exact paths for targeted files.

Test your mapping

After adding the file kind to your mapping configuration, click on the Resync button. When you open the mapping configuration again, you will see real examples of files fetched from your GitLab organization.

This will help you see what data is available to use in your jq expressions. Click on the Test mapping button to test your mapping against the example data.

The structure of the available data is as follows:

Available data example (click to expand)
{
  "file": {
    "file_name": "package.json",
    "file_path": "package.json",
    "size": 780,
    "encoding": "base64",
    "content_sha256": "d4dffc856dcacdaecb283ba9e47288beb6036d7c4ffff7c65f29057d890ecee9",
    "ref": "main",
    "blob_id": "7f8c2fea237a5cf0e1bcc17135c7c8b9e96edd49",
    "commit_id": "3ac75a99f6faa8ce4570368e6db1038c15f17cfc",
    "last_commit_id": "3ac75a99f6faa8ce4570368e6db1038c15f17cfc",
    "execute_filemode": false,
    "content": {
      "name": "my-awesome-project",
      "version": "1.0.0",
      "description": "A sample Node.js project",
      "main": "index.js",
      "scripts": {
        "start": "node index.js",
        "test": "echo \"Error: no test specified\" && exit 1",
        "build": "echo \"Building the project...\"",
        "lint": "eslint ."
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/username/my-awesome-project.git"
      },
      "keywords": [
        "sample",
        "nodejs",
        "project"
      ],
      "author": "Your Name",
      "license": "ISC",
      "bugs": {
        "url": "https://github.com/username/my-awesome-project/issues"
      },
      "homepage": "https://github.com/username/my-awesome-project#readme",
      "dependencies": {
        "@docsly/react": "^1.9.1",
        "@docusaurus/core": "^3.4.0",
        "@docusaurus/plugin-client-redirects": "^3.4.0",
        "@docusaurus/plugin-google-tag-manager": "^3.4.0",
        "@docusaurus/plugin-ideal-image": "^3.4.0",
        "@docusaurus/preset-classic": "^3.4.0",
        "@docusaurus/theme-live-codeblock": "^3.4.0",
        "@easyops-cn/docusaurus-search-local": "^0.44.2",
        "@mdx-js/react": "^3.0.1",
        "@port-labs/docusaurus-plugin-openapi-docs": "^0.0.5",
        "@port-labs/docusaurus-theme-openapi-docs": "^0.0.5",
        "@stackql/docusaurus-plugin-hubspot": "^1.0.1",
        "clsx": "^2.1.1",
        "docusaurus-plugin-hotjar": "^0.0.2",
        "docusaurus-plugin-image-zoom": "^2.0.0",
        "prettier": "^3.3.2",
        "prism-react-renderer": "^2.3.1",
        "react": "^18.3.1",
        "react-dom": "^18.3.1",
        "react-tooltip": "^5.27.1"
      },
      "devDependencies": {
        "@docusaurus/module-type-aliases": "^3.4.0",
        "@docusaurus/tsconfig": "^3.4.0",
        "@docusaurus/types": "^3.4.0",
        "husky": "^9.0.11",
        "pretty-quick": "^4.0.0",
        "typescript": "~5.5.3"
      }
    }
  },
  "repo": {
    "id": 60625101,
    "description": null,
    "name": "pages-test-project",
    "name_with_namespace": "port-labs / pages-test-project",
    "path": "pages-test-project",
    "path_with_namespace": "port-org/pages-test-project",
    "created_at": "2024-08-06T10:31:47.509Z",
    "default_branch": "main",
    "tag_list": [],
    "topics": [],
    "ssh_url_to_repo": "git@gitlab.com:port-org/pages-test-project.git",
    "http_url_to_repo": "https://gitlab.com/port-org/pages-test-project.git",
    "web_url": "https://gitlab.com/port-org/pages-test-project",
    "readme_url": "https://gitlab.com/port-org/pages-test-project/-/blob/main/README.md",
    "forks_count": 0,
    "avatar_url": null,
    "star_count": 0,
    "last_activity_at": "2024-08-14T15:40:56.606Z",
    "namespace": {
      "id": 66136652,
      "name": "port-labs",
      "path": "port-org",
      "kind": "group",
      "full_path": "port-org",
      "parent_id": null,
      "avatar_url": null,
      "web_url": "https://gitlab.com/groups/port-org"
    }
  }
}

Create multiple entities from a single file​

In some cases, we want to parse a single JSON or YAML file and create multiple entities from it. To do this, we can use the itemsToParse key in our mapping configuration.

For example, let's say we want to track or manage a project's dependencies in Port. We'll need to create a package blueprint, with each entity representing a dependency from a package.json file.

The following configuration fetches a package.json file from a specific repository and creates an entity for each dependency in the file, based on the package blueprint:

resources:
  - kind: file
    selector:
      query: 'true'
      files:
        path: 'package.json'
        # Note that in this case we are fetching from a specific repository
        repos:
          - group/my-project
    port:
      itemsToParse: .file.content.dependencies | to_entries
      entity:
        mappings:
          # Since identifier cannot contain special characters, we are using jq to remove them
          identifier: >-
            .item.key + "_" + if (.item.value | startswith("^")) then
            .item.value[1:] else .item.value end
          title: .item.key + "@" + .item.value
          blueprint: '"package"'
          properties:
            package: .item.key
            version: .item.value
          relations: {}

The itemsToParse key is used to specify the path to the array of items you want to parse from the file. In this case, we are parsing the dependencies object from the package.json file.

Once the object is parsed, we can use the item key to refer to each key-value pair within it — where the key is the dependency name, and the value is the version.

This allows us to create an entity for each dependency dynamically.

Multi-document YAML files​

For multi-document YAML files (a single file containing multiple YAML documents separated by ---), .file.content will not resolve to an object, but to an array of objects.

You can use one of these methods to ingest multi-document YAML files:

  1. Use the itemsToParse key to create multiple entities from such a file (see example above).
  2. Map the result to an array property.
Mixed YAML types

If you have both single-document and multi-document YAML files in your repositories, you can use the itemsToParse key like this to handle both cases:

itemsToParse: .file.content | if type== "object" then [.] else . end

Limitations​

  • Currently only files up to 1MB in size are supported.
  • Only JSON and YAML formats are automatically parsed. Other file formats can be ingested as raw files, however, some special characters in the file (such as \n) may be processed and not preserved.
  • Currently only the default branch of the repository is supported.
  • When you omit the repos field, the integration searches across all top-most level groups you have access to. Repositories that are not part of any group are excluded from the search, since the integration uses GitLab's group search API.

For a list of known limitations with GitLab's Advanced Search, see GitLab's Advanced Search documentation.

For practical examples of using the file kind, see the file kind examples page.