-
Notifications
You must be signed in to change notification settings - Fork 0
/
readme.txt
385 lines (357 loc) · 17.4 KB
/
readme.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
Media Process Platform (MPP) module directory description:
MPP : Media Process Platform
MPI : Media Process Interface
HAL : Hardware Abstract Layer
OSAL : Operation System Abstract Layer
Rules:
1. header file arrange rule
a. inc directory in each module folder is for external module usage.
b. module internal header file should be put along with the implement file.
c. header file should not contain any relative path or absolute path, all
include path should be keep in Makefile.
2. compiling system rule
a. for cross platform compiling use cmake as the compiling management system.
b. use cmake out-of-source build, final binary and library will be install to
out/ directory.
3. header file include order
a. MODULE_TAG
b. system header
c. osal header
d. module header
NOTE:
1. Windows support is NOT maintained any more.
2. Mpp support all rockchip chipset now including:
RK29XX/RK30XX/RK31XX
RK3288/RK3368/RK3399
RK3228/RK3229/RK3228H/RK3328
RK3566/RK3568
RV1108/RV1107
RV1109/RV1126
3. Mpp support all format hardware can support except VC1.
4. you can get demo about mpp applied to linux and android.
Liunx : https://github.com/WainDing/mpp_linux_cpp
https://github.com/MUZLATAN/ffmpeg_rtsp_mpp
Android : https://github.com/c-xh/RKMediaCodecDemo
More document can be found at http://opensource.rock-chips.com/wiki_Mpp
---- top
|
|----- build CMake out-of-source build directory
| |
| |----- cmake cmake script directory
| |
| |----- android android build directory
| |
| |----- linux linux build directory
| |
| |----- vc10-x86_64 visual studio 2010 on x86_64 build directory
| |
| |----- vc12-x86_64 visual studio 2013 on x86_64 build directory
|
|----- doc design documents of mpp
|
|----- inc header file for external usage, including
| platform header and mpi header
|
|----- mpp Media Process Platform : mpi function private
| | implement and mpp infrastructure (vpu_api
| | private layer)
| |
| |----- base base components including MppBuffer, MppFrame,
| | MppPacket, MppTask, MppMeta, etc.
| |
| |----- common video codec protocol syntax interface for both
| | codec parser and hal
| |
| |----- codec all video codec parser, convert stream to
| | | protocol structure
| | |
| | |----- inc header files provided by codec module for
| | | external usage
| | |
| | |----- dec
| | | |
| | | |----- dummy decoder parser work flow sample
| | | |
| | | |----- h263
| | | |
| | | |----- h264
| | | |
| | | |----- h265
| | | |
| | | |----- m2v mpeg2 parser
| | | |
| | | |----- mpg4 mpeg4 parser
| | | |
| | | |----- vp8
| | | |
| | | |----- vp9
| | | |
| | | |----- jpeg
| | |
| | |----- enc
| | |
| | |----- dummy encoder controllor work flow sample
| | |
| | |----- h264
| | |
| | |----- h265
| | |
| | |----- jpeg
| |
| |----- hal Hardware Abstract Layer (HAL): modules used in mpi
| | |
| | |----- inc header files provided by hal for external usage
| | |
| | |----- iep iep user library
| | |
| | |----- pp post-processor user library
| | |
| | |----- rga rga user library
| | |
| | |----- deinter deinterlace function module including pp/iep/rga
| | |
| | |----- rkdec rockchip hardware decoder register generation
| | | |
| | | |----- h264d generate register file from H.264 syntax info
| | | |
| | | |----- h265d generate register file from H.265 syntax info
| | | |
| | | |----- vp9d generate register file from vp9 syntax info
| | |
| | |----- vpu vpu register generation library
| | |
| | |----- h263d generate register file from H.263 syntax info
| | |
| | |----- h264d generate register file from H.264 syntax info
| | |
| | |----- h265d generate register file from H.265 syntax info
| | |
| | |----- jpegd generate register file from jpeg syntax info
| | |
| | |----- jpege generate register file from jpeg syntax info
| | |
| | |----- m2vd generate register file from mpeg2 syntax info
| | |
| | |----- mpg4d generate register file from mpeg4 syntax info
| | |
| | |----- vp8d generate register file from vp8 syntax info
| |
| |----- legacy generate new libvpu to include old vpuapi path
| | and new mpp path
| |
| |----- test mpp internal video protocol unit test and demo
|
|----- test mpp buffer/packet component unit test and
| mpp/mpi/vpu_api demo
|
|----- out final release binary output directory
| |
| |----- bin executable binary file output directory
| |
| |----- inc header file output directory
| |
| |----- lib library file output directory
|
|----- osal Operation System Abstract Layer: abstract layer
| | for different operation system
| |
| |----- allocator supported allocator including Android ion and
| | Linux drm
| |
| |----- android google's android
| |
| |----- inc osal header file for mpp modules
| |
| |----- linux mainline linux kernel
| |
| |----- windows microsoft's windows
| |
| |----- test OASL unit test
|
|----- tools coding style format tools
|
|----- utils small util functions
Here is the mpp implement overall framework:
+---------------------------------------+
| |
| ffmpeg / OpenMax / gstreamer / libva |
| |
+---------------------------------------+
+-------------------- MPP ----------------------+
| |
| +-------------------------+ +--------+ |
| | | | | |
| | MPI / MPP | | | |
| | buffer queue manage | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-------------------------+ | | |
| | | | | |
| | codec | | OSAL | |
| | decoder / encoder | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-----------+ +-----------+ | | |
| | | | | | | |
| | parser | | HAL | | | |
| | recoder | | reg_gen | | | |
| | | | | | | |
| +-----------+ +-----------+ +--------| |
| |
+-------------------- MPP ----------------------+
+---------------------------------------+
| |
| kernel |
| RK vcodec_service / v4l2 |
| |
+---------------------------------------+
Here is the Media Process Interface hierarchical structure
MpiPacket and MpiFrame is the stream I/O data structure.
And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and
Android's ion.
This part is learned from ffmpeg.
+-------------------+
| |
| MPI |
| |
+---------+---------+
|
|
v
+---------+---------+
| |
+---+ ctx +---+
| | | |
| +-------------------+ |
| |
v v
+-------+-------+ +-------+-------+
| | | |
| packet | | frame |
| | | |
+---------------+ +-------+-------+
| |
| |
| |
| +---------------+ |
| | | |
+---->+ buffer +<----+
| |
+---------------+
Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer,
MPP will send the stream to codec layer, codec layer parses the stream header
and generates a protocol standard output. This output will be send to HAL to
generate register file set and communicate with hardware. Hardware will complete
the task and resend information back. MPP notify codec by hardware result, codec
output decoded frame by display order.
MPI MPP decoder parser HAL
+ + + + +
| | | | |
| open context | | | |
+----------------> | | | |
| | | | |
| init | | | |
+----------------> | | | |
| | | | |
| | init | | |
| +----------------> | | |
| | | | |
| | | init | |
| | +----------------> | |
| | | | |
| | | | open |
| | +-----------------------------------> |
| | | | |
| decode | | | |
+----------------> | | | |
| | | | |
| | send_stream | | |
| +----------------> | | |
| | | | |
| | | parse_stream | |
| | +----------------> | |
| | | | |
| | | | reg generation |
| | +-----------------------------------> |
| | | | |
| | | | send_regs |
| | +-----------------------------------> |
| | | | |
| | | | wait_regs |
| | +-----------------------------------> |
| | | | |
| | | notify_hw_end | |
| | +----------------> | |
| | | | |
| | get_picture | | |
| +----------------> | | |
| | | | |
| | | get_picture | |
| | +----------------> | |
| | | | |
| flush | | | |
+----------------> | | | |
| | | | |
| | flush | | |
| +----------------> | | |
| | | | |
| | | reset | |
| | +----------------> | |
| | | | |
| close | | | |
+----------------> | | | |
| | | | |
| | close | | |
| +----------------> | | |
| | | | |
| | | close | |
| | +----------------> | |
| | | | |
| | | | close |
| | +-----------------------------------> |
+ + + + +
There are three memory usage modes that decoder can support:
Mode 1: Pure internal mode
In the mode user will NOT call MPP_DEC_SET_EXT_BUF_GROUP control to decoder.
Only call MPP_DEC_SET_INFO_CHANGE_READY to let decoder go on. Then decoder will
use create buffer internally and user need to release each frame they get.
Advantage:
Easy to use and get a demo quickly
Disadvantage:
1. The buffer from decoder may not be return before decoder is close.
So memory leak or crash may happen.
2. The decoder memory usage can not be control. Decoder is on a free-to-run
status and consume all memory it can get.
3. Difficult to implement zero-copy display path.
Mode 2: Half internal mode
This is the mode current mpi_dec_test code using. User need to create
MppBufferGroup according to the returned info change MppFrame.
User can use mpp_buffer_group_limit_config to limit decoder memory usage.
Advantage:
1. Easy to use
2. User can release MppBufferGroup after decoder is closed.
So memory can stay longer safely.
3. Can limit the memory usage by mpp_buffer_group_limit_config
Disadvantage:
1. The buffer limitation is still not accurate. Memory usage is 100% fixed.
2. Also difficult to implement zero-copy display path.
Mode 3: Pure external mode
In this mode use need to create empty MppBufferGroup and import memory from
external allocator by file handle.
On Android surfaceflinger will create buffer. Then mediaserver get file handle
from surfaceflinger and commit to decoder's MppBufferGroup.
Advantage:
1. Most efficient way for zero-copy display
Disadvantage:
1. Difficult to learn and use.
2. Player work flow may limit this usage.
3. May need external parser to get the correct buffer size for the external
allocator.
The required buffer size caculation:
hor_stride * ver_stride * 3 / 2 for pixel data
hor_stride * ver_stride / 2 for extra info
Total hor_stride * ver_stride * 2 will be enough.
For H.264/H.265 20+ buffers will be enough.
For other codec 10 buffers will be enough.