Sentinel Shuttle(English) - skpdi/sentinel-document GitHub Wiki
This is a library managing a format to leave logs based on the defined standard through Sentinel Schema.
You can conveniently leave logs to the column using Setter method.You can download Shuttle from Sentinel Webpage
Shuttle is in charge of a format only. In order to transfer data, Shuttle must be used with Log Collection Tools.
Please refer to the links added to the list below for environment-specific usage guides.
- Client용 Shuttle
- Android (use with Rake Android)
- iOS (use with Rake iOS)
- Javascript (use with Rake Javascript)
- Server용 Shuttle
- Java (use with RakeKafka, LogAgent, Direct-Kafka)
- It can prevent field name error. (IDE auto-fill)
- It supports field value escaping. (\n, \r, other odd ASCII code)
- It prevents field type discordance (If you enter String where Integer is needed, it would not be compiled)
- Create log version automatically
- Log version format: #define within the sheet #version: shuttle version: schema version (concat with ":" without a space)
- Create a log format (JSON/TSV) automatically
- Shuttle is not thread safe. If you need to use multi thread (e.g. Controller), create shuttle every time, and if it is too heavy to create shuttle every time, please use Thread Local, Spring scope.
- Shuttle for Client supports Android, iOS, Javascript version without a dependency
- Shuttle for Server supports a Java version only. It has a dependency. You need to use Shuttle.toString (Not toJSONObject)
| Header / Body | Schema Type (= Hive Table Type) | Java Type (default value in the parentheses) | Javascript | Objective-C | Swift | Note |
|---|---|---|---|---|---|---|
| Header / Body | string | String (null) | var | NSString* (nil) | String? (nil) | |
| Header / Body | long | Long (null) | var | NSNumber* (nil) | Int? (nil) | |
| Header / Body | double | Double (null) | var | NSNumber* (nil) | Double? (nil) | |
| Header Only | fixed string(n) | String (null) | var | NSString* (nil) | String? (nil) | |
| Body Only | list<long> | List<Long> (new ArrayList<Long>) | var | NSMutableArray* (nil) | [Int]? (nil) | |
| Body Only | list<double> | List<Double> (new ArrayList<Double>) | var | NSMutableArray* (nil) | [Double]? (nil) | |
| Body Only | list<string> | List<String> (new ArrayList<String>) | var | NSMutableArray* (nil) | [String]? (nil) | |
| Body Only | map<string,long> | Map<String, Long> (new LinkedHashMap<String, Long>) | var | NSMutableDictionary* (nil) | [String:Int]? (nil) | |
| Body Only | map<string,double> | Map<String, Double> (new LinkedHashMap<String, Double>) | var | NSMutableDictionary* (nil) | [String:Double]? (nil) | |
| Body Only | map<string,string> | Map<String, String> (new LinkedHashMap<String, String>) | var | NSMutableDictionary* (nil) | [String:String]? (nil) | |
| Body Only | json | JSONObject (null) | var | NSMutableDictionary* (nil) | [String:Any]? (nil) | encryption and verification will not be supported |
If you use JSON type for Body column type, the errors like below might occur.
In this case, the body, which is 1 depth originally, will be more than 2 depth. Encryption would not be supproted for the value which json is more than 2 depth Verification would not be supported for the value which Json is more than 2 depth
You should be aware of those issues and discuss log encryption or verification issues with DAS/DA
For your information, json value with several depths can be extracted using get_json_object() function in Hive.
| Field: Header / Body | Shuttle: Client / Server | INPUT: none / null / ""
|
STORAGE: Kafka, HDFS | OUTPUT | note | migration |
|---|---|---|---|---|---|---|
| Header | Client | none (default value, nothing has been entered) | Kafka (intermediate stage) | "" |
"" represents an empty value |
Later, it will be changed to JSONObject.NULL If JSONObject.NULL represents empty values, logs trasferred to client shuttle would be changed to an empty string '' when saving "" in TSV at HDFS. |
| Header | Client | null |
Kafka (Intermediate stage) | "" |
"" represents an empty value |
Later, it will be changed to JSONObject.NULL. JSONObject.NULL represents empty values, logs transferred to the client shuttle would be changed to an empty string '' when saving ""in TSV at HDFS. |
| Header | Server | null |
Kafka (Intermediate stage) | "" |
"" represents an empty value |
Server log will be TSV format since the intermediate stage. |
| Header | Server | none (default value, nothing has been entered) | Kafka (Intermediate stage) | "" |
"" represents an empty value |
Server logs will be TSV format since the intermediate stage |
| Header | Header |
"" (empty string) |
Kafka (Intermediate stage) | "" |
||
| Header | Header | ALL | HDFS (The final stage) | '' |
As logs are saved in TSV format, "" in the intermediate stage becomes '' (empty string, null value) |
|
| Body | Client | none | ALL | Delete the key | Delete the key if there is no value in the body | Later, it will be changed to JSONObject.NULL
|
| Body | Client | null |
ALL | Delete the key | Delete the key to if there is no value in the body | Later, it will be changed to JSONObject.NULL
|
| Body | Server | none | ALL | Delete the key | Delete the key if there is no value in the body | |
| Body | Server | null |
ALL | Delete the key | Delete the key if there is no value in the body | |
| Body | ALL | "" |
ALL | "" |
To sum up, be aware that even though null value can be used to express empty value in Shuttle, the empty value will be represented differently in Header (TSV) and Body (JSON), and the value will also be different based on saved location (Kafka or HDFS).