How to create a single segment stream for ByteWriter?

[junglie85]

The ByteWriter requires a single segment stream as per a comment in the docs:

// assuming scope:myscope, stream:mystream exist.
// notice that this stream should be a fixed sized single segment stream
let stream = ScopedStream::from("myscope/mystream");

I can see in pravega-admin that after creating a scope and a stream using pravega-cli, I have 4 segments:

> controller describe-stream my-scope my-stream
Successful REST request.
{
  "scopeName": "my-scope",
  "streamName": "my-stream",
  "scalingPolicy": {
    "type": "FIXED_NUM_SEGMENTS",
    "minSegments": 4
  },
  "tags": [],
  "timestampAggregationTimeout": 0,
  "rolloverSizeBytes": 0
}

I cannot find anywhere in the docs or examples, either for the Rust client or Pravega generally, how to set the segment count to 1. Please could you let me know how to do this, and perhaps also update the docs also?

[thekingofcity]

PravegaCtl provides a way to create a stream with a specific segment count. It is not listed in the doc but here at pravega-client-rust/pravegactl/src/main.rs#L113.

Also, both Python and NodeJS bindings allow users to create a stream with a specific segment count.
https://github.com/pravega/pravega-client-rust/blob/master/python/src/stream_manager.rs#L284
https://github.com/pravega/pravega-client-rust/blob/master/nodejs/stream_manager.ts#L299

[junglie85]

Thanks @thekingofcity that's helpful.

I'm actually trying to create a thin wrapper around the client that I can call from a C++ application. I'm trying to get a simple write and read working in Rust first. I couldn't get the event API to work - it kept hanging so wanted to try the byte API. Both of these are using a standalone server (0.13.0) with JDWP changed to 9012 (default is already in use/blocked on PC).

Any idea why this example hangs after writing the payload (if seek to tail is uncommented, it hangs on this call)?

use anyhow::anyhow;
use pravega_client::client_factory::ClientFactoryAsync;
use pravega_client_config::ClientConfigBuilder;
use pravega_client_shared::ScopedStream;
use tokio::runtime::Handle;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ClientConfigBuilder::default()
        .controller_uri("localhost:9090")
        .build()
        .map_err(|s| anyhow!(s))?;
    let handle = Handle::current();
    let client_factory = ClientFactoryAsync::new(config, handle);

    let stream = ScopedStream::from("my-scope/my-stream");

    let mut byte_writer = client_factory.create_byte_writer(stream.clone()).await;
    println!("created writer");
    // byte_writer.seek_to_tail().await;
    // println!("seeked to tail");

    let payload = "Hello, Pravega!".to_string().into_bytes();

    byte_writer.write(&payload).await?;
    println!("wrote payload");
    byte_writer.flush().await?;
    println!("flushed writer");
    // byte_writer.truncate_data_before(4).await?;
    // byte_writer.seal().await?;

    let mut byte_reader = client_factory.create_byte_reader(stream).await;
    println!("created reader");

    // let offset = byte_reader.current_head().await?;
    let mut buf: Vec<u8> = vec![0; 4];
    let _size = byte_reader.read(&mut buf).await?;
    println!("{:?}", buf);

    Ok(())
}

Outputs:

created writer
wrote payload

[thekingofcity]

Not quite familiar with byte things but one thing that might block read/write is a bug mentioned in #373. Are you working on Windows?

Another thing that might help is that there is a C-compatible dynamic library in #377 and you may have a try if you like.

[junglie85]

Funny, yes, I am on Windows, and I'd just gone back through all the releases and it works on 0.9.1, fails on 0.10.2 (didn't try the inbetween versions).

I'll have a look at the Go binding. I assume building the bindings is what generates the c library?

[thekingofcity]

From the README yes it should generate a c library. @vangork @CharlieLChen

How to use pravega go client

1. build rust dynamic library
   cd golang
   cargo build --release
   cd .. # cd to rust project root directory
   mv ./target/release/libpravega_client_c.so /usr/lib

[junglie85]

I actually couldn't get the build to succeed.

PS C:\Dev\vangork-pravega-client-rust\golang> cargo build --release
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   C:\Dev\vangork-pravega-client-rust\pravegactl\Cargo.toml
workspace: C:\Dev\vangork-pravega-client-rust\Cargo.toml
   Compiling pravega_client_c v0.3.2 (C:\Dev\vangork-pravega-client-rust\golang)
warning: unused return value of `Box::<T>::from_raw` that must be used
   --> golang\src\stream_manager.rs:117:13
    |
117 |             Box::from_raw(manager);
    |             ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: call `drop(Box::from_raw(ptr))` if you intend to drop the `Box`
    = note: `#[warn(unused_must_use)]` on by default

warning: unused return value of `Box::<T>::from_raw` that must be used
   --> golang\src\stream_manager.rs:271:13
    |
271 |             Box::from_raw(rg);
    |             ^^^^^^^^^^^^^^^^^
    |
    = note: call `drop(Box::from_raw(ptr))` if you intend to drop the `Box`

warning: unused return value of `Box::<T>::from_raw` that must be used
   --> golang\src\stream_writer.rs:125:9
    |
125 |         Box::from_raw(writer);
    |         ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: call `drop(Box::from_raw(ptr))` if you intend to drop the `Box`

warning: unused return value of `Box::<T>::from_raw` that must be used
  --> golang\src\stream_reader_group.rs:64:13
   |
64 |             Box::from_raw(reader);
   |             ^^^^^^^^^^^^^^^^^^^^^
   |
   = note: call `drop(Box::from_raw(ptr))` if you intend to drop the `Box`

warning: unused return value of `Box::<T>::from_raw` that must be used
  --> golang\src\stream_reader.rs:57:13
   |
57 |             Box::from_raw(slice);
   |             ^^^^^^^^^^^^^^^^^^^^
   |
   = note: call `drop(Box::from_raw(ptr))` if you intend to drop the `Box`

error: linking with `link.exe` failed: exit code: 1120
  |
  = note: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX64\\x64\\link.exe" "/DEF:C:\\Users\\JUNGLI~1\\AppData\\Local\\Temp\\rustcoTWgoW\\lib.def" "/NOLOGO" "C:\\Users\\JUNGLI~1\\AppData\\Local\\Temp\\rustcoTWgoW\\symbols.o" "C:\\Dev\\vangork-pravega-client-rust\\target\\release\\deps\\pravega_client_c.pravega_client_c.14ad4302-cgu.0.rcgu.o" "/LIBPATH:C:\\Dev\\vangork-pravega-client-rust\\target\\release\\deps" "/LIBPATH:C:\\Users\\junglie85\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\windows_x86_64_msvc-0.42.2\\lib" "/LIBPATH:C:\\Dev\\vangork-pravega-client-rust\\target\\release\\build\\ring-d3fcac173c8f3378\\out" "/LIBPATH:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\JUNGLI~1\\AppData\\Local\\Temp\\rustcoTWgoW\\libring-6c59f5b035e534f5.rlib" "C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-58e2de43e9b1a814.rlib" "advapi32.lib" "bcrypt.lib" "windows.lib" "advapi32.lib" "cfgmgr32.lib" "credui.lib" "fwpuclnt.lib" "gdi32.lib" "kernel32.lib" "msimg32.lib" "ntdll.lib" "ole32.lib" "opengl32.lib" "runtimeobject.lib" "secur32.lib" "user32.lib" "winspool.lib" "ws2_32.lib" "ntdll.lib" "windows.lib" "legacy_stdio_definitions.lib" "kernel32.lib" "advapi32.lib" "userenv.lib" "kernel32.lib" "ws2_32.lib" "bcrypt.lib" "msvcrt.lib" "legacy_stdio_definitions.lib" "/NXCOMPAT" "/LIBPATH:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "/OUT:C:\\Dev\\vangork-pravega-client-rust\\target\\release\\deps\\pravega_client_c.dll" "/OPT:REF,ICF" "/DLL" "/IMPLIB:C:\\Dev\\vangork-pravega-client-rust\\target\\release\\deps\\pravega_client_c.dll.lib" "/DEBUG" "/NATVIS:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\junglie85\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note:    Creating library C:\Dev\vangork-pravega-client-rust\target\release\deps\pravega_client_c.dll.lib and object C:\Dev\vangork-pravega-client-rust\target\release\deps\pravega_client_c.dll.exp
          pravega_client_c.pravega_client_c.14ad4302-cgu.0.rcgu.o : error LNK2019: unresolved external symbol ackOperationDone referenced in function _ZN16pravega_client_c13stream_writer12StreamWriter11run_reactor28_$u7b$$u7b$closure$u7d$$u7d$17hc9aa075f9bcdfbe3E
          C:\Dev\vangork-pravega-client-rust\target\release\deps\pravega_client_c.dll : fatal error LNK1120: 1 unresolved externals


warning: `pravega_client_c` (lib) generated 5 warnings
error: could not compile `pravega_client_c` due to previous error; 5 warnings emitted

Maybe that'd be better off being raised as an error in the fork @vangork ?

[CharlieLChen]

we never build the go-binding-client on windows, maybe you can try to build it again on Linux

[junglie85]

I need this to work on Windows, so that's out of the question. I've gone with my own FFI wrapper with the basic functionality I need, but appreciate the suggestion of trying the Go client.

[thekingofcity]

Great to hear about your achievements. Looks like all your problems are solved in this issue :)