[data processing] what is lineage Analysis

vividime-Club Show all floors Published on 2024-12-10 16:35:10 |Reading mode print Previous Topic Next Topic
1
Unresolved

【data processing】 what is lineage Analysis

1179 1
Lineage Analysis                  
reply

Using props report

Exciting comments1

vividime-Club Show all floors Published on 2024-12-10 16:35:17
1. Description of the page
The Lineage Analysis page allows you to perform a number of operations to view the lineage analysis, which is accomplished through buttons on the page with the following page parameters:
Parameter
Description

Resource Type
Resource Type include Dashboard, Dataset, and Experiment. You can select one of the types you need to view in the drop-down selection box.

Select Resource
Based on the selected resource type, list the resources that match, currently all resources will be displayed, select the resources you want to view through the drop-down list.

Search
Once the resource is selected, click on the Search button to query the results of the resource's lineage analysis.

Display Impact Analysis
When " Display Impact Analysis " is checked, information on the impact data of other resources on the current resource is displayed in the organization chart below. For example, Dashboard B is hyperlinked to Dashboard A.

Display Lineage Analysis
When "Show Lineage Analysis" is checked, information about the current resource's lineage data is displayed in the organization chart below.

Display Component Nodes
After checking the " Display Component Nodes " box, the component node information of the current resource will be displayed in the organization chart below.

Display Hyperlink Dashboard
When " Display Hyperlink Dashboard " is checked, the organization chart below will show the hyperlink dashboard information of the current resource.

Refresh Lineage Data
Clicking the Refresh button will refresh the page after the resource lineage analysis has changed.

On the Lineage Analysis page, different types of nodes are distinguished by color, as shown in the following figure.
Refer to the table below for supporting lineage analysis.
Lineage Analysis Type
Reference Chart

Datasets > SQL Statements > Connections

Datasets > Database Tables > Connections

Dataset > Marketplace Folder > Timed Tasks > Dataset > Datasheet or SQL Statement > Connection

Dataset > Timed Tasks > Datasets > Database Tables > Connections

Dataset > Timed Tasks > Datasets > SQL Statements > Connections

Datasets > Datasets

Dashboards > Hyperlink Dashboards

Complex Dashboards > Complex Dashboards

Complex Dashboards > Web

Complex Dashboards > Dashboards

Mouse click on the node of the lineage analysis will show the details of the current node on the right side, refer to the table below for details.
Node Type
Details

Complex dashboards, dashboards, components, experiments, bazaar folders, database tables
Include the name, type, and path, and refer to the experimental node in the figure below.

Component
Includes the name, type, and binding fields of the component.

SQL statement
Includes SQL statements, SQL fields in database tables .

Dataset
Includes name, type, path, bound fields. Click Open Dataset in the lower right corner to jump to the Dataset module to view and edit.
The Excel dataset also includes the Excel file path.
The dataset corresponding to the data model node is extracted and the extraction icon and extraction information are displayed.
The data model node is extracted to display the extraction icon and extraction information.

Timed task
Includes name, type, path, and last trigger time.

2. Parameter Configuration
If it is a database system, it can be assigned to db.properties or to bi.properties in bihome under database. Read order db.properties > bi.properties.
If it is a file system, it is configured into bi.properties.
•data.lineage.init // Default is true, when configured to true, the product will first parse the resources under bihome when it starts and store their lineage to the database.
•data.lineage.clear // Default false, this property takes effect when data.lineage.init=true, when configured to true, the data in the lineage table in the database will be cleared when starting the product. The clearing action occurs before the initialization write. You can configure this property if you want to regenerate the lineage data.
•data.lineage.save.column // Default false, no column information is displayed in the dataset's lineage graph; when configured to true, column information is displayed in the component/dataset node.
An example of displaying field information is as follows:
•data.lineage.cache.max.size // buffer pool size, default 3000. control the memory occupation by controlling the size of the buffer pool to reduce the continuous CPU occupation by read and write threads.
•data.lineage.write.max.size // The number of rows to be written to the database, default 1000. a commit operation is performed when the specified number is reached. Reduces the frequency of commits.
•data.lineage.debug //default false, configured to true to display lineage-related log messages in log management.
•data.lineage.sql.debug //default is false, when configured as true, print the time of SQL execution related to lineage, generally used for debugging, will generate a lot of logs when turned on.


reply

Using props report

Advanced mode
You need to log in before you can reply to the post login | Free registration

© 2024 VIVIDATA PTE. LTD. All Rights Reserved. Privacy Statement