Web-based Cloud Gaming service for Retro Game

CloudRetro

Build Latest release

Open-source Cloud Gaming Service For Retro Games
Video demo: https://www.youtube.com/watch?v=GUBrJGAxZZg
Technical wrapup: https://webrtchacks.com/open-source-cloud-gaming-with-webrtc/
CloudMorph: https://github.com/giongto35/cloud-morph: My current focus on generic solution for cloudgaming

Introduction

CloudRetro provides an open-source cloud gaming platform for retro games. It started as an experiment for testing cloud gaming performance with WebRTC and libretro, and now it aims to deliver the most modern and convenient gaming experience through the technology.

Theoretically, in cloud gaming, games are run on remote servers and media are streamed to the player optimally to ensure the most comfortable user interaction. It opens the ability to play any retro games on web-browser directly, which are fully compatible with multi-platform like Desktop, Android, IOS.

Announcement

(Currently, I'm working on CloudMorph: It offers more generic solution to run any offline games/application on browser in Cloud Gaming approach: https://github.com/giongto35/cloud-morph)

Try the service at

Single play: http://cloudretro.io
Direct play an existing game: Pokemon Emerald

*In ideal network condition and less resource contention on servers, the game will run smoothly as in the video demo. Because I only hosted the platform on limited servers in US East, US West, Eu, Singapore, you may experience some latency issues + connection problem. You can try hosting the service following the instruction the next section to have a better sense of performance.

Screenshot Screenshot
screenshot screenshot
screenshot screenshot

Feature

  1. Cloud gaming: Game logic and storage is hosted on cloud service. It reduces the cumbersome of game initialization. Images and audio are streamed to user in the most optimal way using advanced encoding technology.
  2. Cross-platform compatibility: The game is run on web browser, the most universal built-in app. No console, plugin, external app or devices are needed.
  3. Emulator agnostic: The game can be played directly without any extra effort to set up the gaming emulator or platform.
  4. Collaborate gameplay: Follow the idea of crowdplay(TwitchPlaysPokemon), multiple players can play the same game together by addressing the same deeplink. The game experience is powered by cloud-gaming, so the game is much smoother. Check CrowdPlay section
  5. Online multiplayer: The first time in history, you can play multiplayer on Retro games online. You can try Samurai Showndown with 2 players for fighting game example.
  6. Horizontally scaled: The infrastructure is designed to be able to scale under high traffic by adding more instances.
  7. Cloud storage: Game state is storing on online storage, so you can come back and continue playing your incomplete game later.

Development environment

Install Golang https://golang.org/doc/install . Because the project uses GoModule, so it requires Go1.11 version.

Install Dependencies

# Ubuntu / Windows (WSL2)
apt-get install -y make gcc pkg-config libvpx-dev libopus-dev libopusfile-dev libsdl2-dev

# MacOS
brew install libvpx pkg-config opus opusfile sdl2

# Windows (MSYS2)
pacman -Sy --noconfirm --needed git make mingw-w64-x86_64-{gcc,pkg-config,dlfcn,libvpx,opusfile,SDL2}

Because the coordinator and workers need to run simultaneously. Workers connect to the coordinator.

  1. Script
  • make dev.run
  • The scripts spawns 2 processes one in the background and one in foreground
  1. Manual
  • Need to run coordinator and worker separately in two session
  • go run cmd/coordinator/main.go - spawn coordinator
  • go run cmd/worker/main.go --coordinatorhost localhost:8000 - spawn workers connecting to coordinator

Additionally, you may install and configure an X Server display in order to be able to run OpenGL cores. See the docker-compose.yml file for Xvfb example config.

Run with Docker

Use makefile script: make dev.run-docker or Docker Compose directly: docker-compose up --build (CLOUD_GAME_GAMES_PATH is env variable for games on your host). It will spawn a docker environment and you can access the service on localhost:8000.

Configuration

The configuration parameters are stored in the configs/config.yaml file which is shared for all application instances on the same host system. It is possible to specify individual configuration files for each instance as well as override some parameters, for that purpose, please refer to the list of command-line options of the apps.

Technical Document

High level Worker internal
screenshot screenshot

FAQ

Crowd Play, play game together

By clicking these deep link, you can join the game directly and play it together with other people.

And you can host the new game by yourself by accessing cloudretro.io and click "share" button to generate a deeplink to your current game.


Synchronize a game session on multiple devices

Contribution

We are very much thankful to everyone who contributes to the project:

Credits

Art

Main Contributor

Sergey Stepanov
https://github.com/sergystepanov/

Author

Nguyen Huu Thanh
https://www.linkedin.com/in/huuthanhnguyen/

Tri Dang Minh
https://trich.im

Comments
  • Add aspect ratio support

    Add aspect ratio support

    Added proper nearest neighbour up/downscaling support with aspect ratio calculation. Now it sets the initial viewport (output image) resolution to the base resolution from the loaded core upscaled by 2 to remove resampling artefacts (as on pictures below). So the manual resolution config for each core now removed completely. A Sony BIOS intro now is being shown properly. Need some testing.

    Before image

    After image

  • Test Google Colab

    Test Google Colab

    I have managed to get it to work in Google Colab, as far as Cloud-Game is running, but since it stays running, I can't run ngrok to make localhost public.

    image

    image

    Can you help me finish the cell?

    https://gist.github.com/ImanCol/2928cc0870efcca8cd0041cd1fa4975f

    Is there another way to access localhost?

    Google Colab: Distributor ID: Ubuntu Description: Ubuntu 18.04.5 LTS Release: 18.04 Codename: bionic

  • YCbCr corrections

    YCbCr corrections

    Summary

    To try to correct conversion artifacts of YUV 4:2:0 images used in compressed video.

    Info

    In both video encoders we use YUV 4:2:0 encoded frames. YUV is a class of pixel formats used in video applications. YUV colors are represented with one "luminance" component called Y (equivalent to grey scale) and two "chrominance" components, called U (blue projection) and V (red projection) respectively. YUV can also be referred to as YCbCr, although the terms mean slightly different things.

    Planar formats Planar formats use separate matrices for each of the 3 color components. In other words, there is one table of luminance pixel values, and two separate tables for the chrominance components. This segregated representation in memory of pixels is more convenient for video coding.

    YUV 4:2:0 (I420/J420/YV12)

    I420 has the luma "luminance" plane Y first, then the U chroma plane and last the V chroma plane. The two chroma planes (blue and red projections) are sub-sampled in both the horizontal and vertical dimensions by a factor of 2. That is to say, for a 2×2 square of pixels, there are 4 Y samples but only 1 U sample and 1 V sample. This format requires 4×8+8+8=48 bits per 4 pixels, so its depth is 12 bits per pixel. I420 is by far the most common format.

    A graphical illustration: Each letter represents one bit.

    For a single I420 pixel: YYYYYYYY UU VV For a 50-pixel I420 frame: YYYYYYYY×50 UU×50 VV×50 (or Y×8×50 U×2×50 V×2×50 for short) For an n-pixel I420 frame: Y×8×n U×2×n V×2×n

    J420 J420 is exactly like I420, but with a full range ("digital", 0-255) luma (Y) component instead of limited range ("analog", 16-240). The chroma planes are exactly the same as in I420.

    YV12 YV12 is exactly like I420, but the order of the U and V planes is reversed. In the name, "YV" refers to the plane order: Y, then V (then U). "12" refers to the pixel depth: 12-bits per pixel as for I420.

    Color ranges BT.601 studio (TV), full (PC) swing

    Prior to the development of fast SIMD floating-point processors, most digital implementations of RGB → Y′UV used integer math, in particular fixed-point approximations. Approximation means that the precision of the used numbers is limited in favor of improved computation speeds.

    Y′ values are conventionally shifted and scaled to the range [16, 235] (referred to as studio swing or "TV levels") rather than using the full range of [0, 255] (referred to as full swing or "PC levels"). The value 235 accommodates a maximal black-to-white overshoot of 255 − 235 = 20, or 20 / (235 − 16) = 9.1%, which is slightly larger than the theoretical maximal overshoot (Gibbs phenomenon) of about 8.9% of the maximal step. The toe-room is smaller, allowing only 16 / 219 = 7.3% overshoot, which is less than the theoretical maximal overshoot of 8.9%. This is why 16 is added to Y′ and why the Y′ coefficients in the basic transform sum to 220 instead of 255. U and V values, which may be positive or negative, are summed with 128 to make them always positive, giving a studio range of 16–240 for U and V. (These ranges are important in video editing and production since using the wrong range will result either in an image with "clipped" blacks and whites, or a low-contrast image.)

    BT.601 (studio/full) swing. For getting the traditional "studio-swing" 8-bit representation of Y′UV for SDTV/BT.601 the following operations can be used:

    1. Basic transform from 8-bit RGB to 16-bit values (Y′: unsigned, U/V: signed, matrix values got rounded so that the later-on desired Y′ range of [16..235] and U/V range of [16..240] is reached):
    Y'  |  66 129  25 | R |
    U = | -38 -74 112 | G |
    V   | 112 -94 -18 | B |
    
    1. Scale down (">>8") to 8 bits with rounding ("+128") (Y′: unsigned, U/V: signed):
    Yt'= (Y'+ 128) >> 8
    Ut = (U + 128) >> 8
    Vt = (V + 128) >> 8
    
    1. Add an offset to the values to eliminate any negative values (all results are 8-bit unsigned):
    Yu'= Yt' + 16
    Uu = Ut + 128
    Vu = Vt + 128
    

    Full swing for BT.601. For getting a "full-swing" 8-bit representation of Y′UV for SDTV/BT.601 can be used with the same steps except for use of different weights on the first step.

    Our implementation uses studio swing integer math without SIMD. Generally speaking, full swing has the advantage of more picture information with less color banding or quantization errors but in our case, it's easier to stick with the studio range because VP8 codec doesn't support the full range (VP9 does) and we would have to implement both algorithms with some conditional switch on each codec. It is now accepted that SD content uses the BT.601 colorspace and HD uses BT.709.

    The problem

    We have YUV artifacts after RGB image conversion. So either YUV 4:2:0 converter is trash by itself or 4:2:0 is not meant for small images.

    RGB/YUV (edited) image comparison...

    image

    RGB/YUV raw image comparison...

    download (7) download (5)

    VP8 encoded clip...

    feature007

    Direct RGB/YUV UV (4:4:4/4:2:0) comparison...

    image image

    image image

    Solutions

    1. Since blue and red channels are very crippled it's possible just to upscale original frames by a factor of 2 or more.

    2. Rework the existing YUV-converter in a way that it could interpolate chrominance values more precisely. In the current algorithm, we calculate a single chroma value for all 2x2 RGB/luma values just based on the first pixel:

    XO  X   XO  X
    
    X   X   X   X
    

    This will lead to aliasing and color artifacts, instead, it is better to calculate a new chroma value for the average color between these 4 RGB pixels:

    X   X   X   X
      O       O
    X   X   X   X
    
    1. Try to use YUV 422 or 444 if codecs can support them. These formats should have fewer or no chrominance conversion artifacts but will reduce compression effectiveness as well as limit WebRTC client support.

    Implementation notes

    The first solution can be implemented just by changing one configuration param: https://github.com/giongto35/cloud-game/blob/90943f0cc4b916ccb959b20b32f755d231b64a24/configs/config.yaml#L64-L66 One disadvantage of this method is that change will scale output images for every emulator which could be heavy on the performance side.

    The second one is implemented in the new PR. Added optimized multithreaded YUV converter with 2x2 chroma color estimation. Try sqrt -- slow.

    The third solution seems to be hard to implement for both encoders because of difficulties in support of these image compression types.

    Literature

    001 / VideoLan Wiki (site) 002 / Wikipedia YCbCr (site)

    Tools used

    001 / yuvtoolkit (site) 002 / rawpixels.net (site)

    Originally posted by @sergystepanov in https://github.com/giongto35/cloud-game/issues/285#issuecomment-784655254

  • WebRTC won't start in Firefox (Windows 10)

    WebRTC won't start in Firefox (Windows 10)

    Is it just me or the app still isn't working in Firefox? It says in the browser console: ICE failed, your TURN server appears to be broken, see about:webrtc for more details Is this suppose to work now without additional configuration?

    FF: 75.0 Windows

    I0427 23:52:40.728461    8564 handlers.go:161] Coordinator: A user is connecting...
    I0427 23:52:40.729464    8564 browser.go:35] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Generated worker ID
    I0427 23:52:40.731461    8564 browser.go:30] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Get Room  Zone  From URL /ws?room_id=&zone=
    I0427 23:52:40.731461    8564 browser.go:35] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Send sync[http://localhost:9000/echo]http://localhost:9000/echo
    I0427 23:52:40.755463    4088 worker.go:75] &{0xc000068aa0 0xc000574000 {} 0x4a3240 false false false false 0xc0000608c0 {0xc0000d0700 map[] false false} map[Access-Control-Allow-Origin:[*]] true 0 -1 0 false false [] 0 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 0] [0 0 0] 0xc0004a0070 0} echo
    I0427 23:52:40.789460    8564 browser.go:35] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Latency mapmap[0xc000180480:41]
    I0427 23:52:40.826462    8564 browser.go:30] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Received initwebrtc request -> relay to worker: 6a1a220d-5c62-4f5b-a8b4-eea0c246d828
    I0427 23:52:40.828462    4088 coordinator.go:57] Received a request to createOffer from browser via coordinator
    I0427 23:52:40.828462    4088 webrtc.go:111] === StartClient ===
    I0427 23:52:40.933462    4088 webrtc.go:131] Add video track
    I0427 23:52:40.933462    4088 webrtc.go:142] Add audio track
    I0427 23:52:40.934460    4088 webrtc.go:200] Created Offer
    I0427 23:52:40.934460    4088 webrtc.go:181] OnIceCandidate: candidate:foundation 1 udp 2130706431 192.168.234.33 63191
    typ host
    I0427 23:52:40.935466    4088 coordinator.go:89] Start peerconnection 5a74aa09-9b6d-4c7c-9bff-941b63c24e05
    I0427 23:52:40.935466    8564 worker.go:41] Worker 6a1a220d-5c62-4f5b-a8b4-eea0c246d828] Received IceCandidate from worker -> relay to browser
    I0427 23:52:40.935466    4088 webrtc.go:181] OnIceCandidate: candidate:foundation 1 udp 2130706431 192.168.1.121 63192 typ host
    I0427 23:52:40.935466    8564 browser.go:35] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Received SDP from worker -> sending back to browser
    I0427 23:52:40.936462    4088 webrtc.go:181] OnIceCandidate: candidate:foundation 1 udp 2130706431 192.168.56.1 63193 typ host
    I0427 23:52:40.936462    8564 worker.go:41] Worker 6a1a220d-5c62-4f5b-a8b4-eea0c246d828] Received IceCandidate from worker -> relay to browser
    I0427 23:52:40.936462    4088 webrtc.go:181] OnIceCandidate: candidate:foundation 1 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194
    I0427 23:52:40.937463    8564 worker.go:41] Worker 6a1a220d-5c62-4f5b-a8b4-eea0c246d828] Received IceCandidate from worker -> relay to browser
    I0427 23:52:40.938462    8564 worker.go:41] Worker 6a1a220d-5c62-4f5b-a8b4-eea0c246d828] Received IceCandidate from worker -> relay to browser
    I0427 23:52:40.939464    8564 worker.go:41] Worker 6a1a220d-5c62-4f5b-a8b4-eea0c246d828] Received IceCandidate from worker -> relay to browser
    I0427 23:52:40.975462    8564 browser.go:35] Browser 5a74aa09-9b6d-4c7c-9bff-941b63c24e05] Received browser answered SDP -> relay to worker
    I0427 23:52:40.982463    4088 coordinator.go:106] Received answer SDP from browser
    I0427 23:52:40.984465    4088 webrtc.go:233] Set Remote Description
    I0427 23:52:40.985462    4088 webrtc.go:165] ICE Connection State has changed: checking
    I0427 23:52:50.985852    4088 webrtc.go:165] ICE Connection State has changed: failed
    
    FF log
    (registry/INFO) insert 'ice' (registry) succeeded: ice
    (registry/INFO) insert 'ice.pref' (registry) succeeded: ice.pref
    (registry/INFO) insert 'ice.pref.type' (registry) succeeded: ice.pref.type
    (registry/INFO) insert 'ice.pref.type.srv_rflx' (UCHAR) succeeded: 0x64
    (registry/INFO) insert 'ice.pref.type.peer_rflx' (UCHAR) succeeded: 0x6e
    (registry/INFO) insert 'ice.pref.type.host' (UCHAR) succeeded: 0x7e
    (registry/INFO) insert 'ice.pref.type.relayed' (UCHAR) succeeded: 0x05
    (registry/INFO) insert 'ice.pref.type.srv_rflx_tcp' (UCHAR) succeeded: 0x63
    (registry/INFO) insert 'ice.pref.type.peer_rflx_tcp' (UCHAR) succeeded: 0x6d
    (registry/INFO) insert 'ice.pref.type.host_tcp' (UCHAR) succeeded: 0x7d
    (registry/INFO) insert 'ice.pref.type.relayed_tcp' (UCHAR) succeeded: 0x00
    (registry/INFO) insert 'stun' (registry) succeeded: stun
    (registry/INFO) insert 'stun.client' (registry) succeeded: stun.client
    (registry/INFO) insert 'stun.client.maximum_transmits' (UINT4) succeeded: 7
    (registry/INFO) insert 'ice.trickle_grace_period' (UINT4) succeeded: 5000
    (registry/INFO) insert 'ice.tcp' (registry) succeeded: ice.tcp
    (registry/INFO) insert 'ice.tcp.so_sock_count' (INT4) succeeded: 0
    (registry/INFO) insert 'ice.tcp.listen_backlog' (INT4) succeeded: 10
    (registry/INFO) insert 'ice.tcp.disable' (char) succeeded: \000
    (generic/ERR) UDP socket error:Internal error at Z:/task_1585933993/build/src/dom/network/UDPSocketParent.cpp:268 this=00000270B834C000
    (ice/ERR) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): failed to find default addresses
    (generic/ERR) UDP socket error:Internal error at Z:/task_1585933993/build/src/dom/network/UDPSocketParent.cpp:268 this=00000270B834E000
    (ice/ERR) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): failed to find default addresses
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_0 - 01f5805e:45385da4adf81024c3fcd93ae27bfec0
    (ice/INFO) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) no such component for candidate candidate:foundation 2 udp 2130706431 192.168.234.33 63191 typ host generation 0
    (ice/INFO) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) no such component for candidate candidate:foundation 2 udp 2130706431 192.168.1.121 63192 typ host generation 0
    (ice/INFO) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) no such component for candidate candidate:foundation 2 udp 2130706431 192.168.56.1 63193 typ host generation 0
    (ice/INFO) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) no such component for candidate candidate:foundation 2 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    (generic/ERR) UDP socket error:Internal error at Z:/task_1585933993/build/src/dom/network/UDPSocketParent.cpp:268 this=00000270B834E000
    (ice/ERR) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): failed to find default addresses
    (generic/ERR) UDP socket error:Internal error at Z:/task_1585933993/build/src/dom/network/UDPSocketParent.cpp:268 this=00000270B8350C00
    (ice/ERR) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): failed to find default addresses
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_0 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_3 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_4 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_0 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_3 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_4 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_0 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_3 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_4 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_0 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_3 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    (ice/WARNING) ICE(PC:1588020760802000 (id=10737418241 url=http://localhost:8000/)): peer (PC:1588020760802000 (id=10737418241 url=http://localhost:8000/):default) has no stream matching stream PC:1588020760802000 (id=10737418241 url=http://localhost:8000/) transport-id=transport_4 - 9d8a2bf7:e0e1a833f36d9e0fe0012984ec6dd249
    +++++++ END ++++++++
    
    SDP
    SDP
    Local SDP (Offer)
    v=0
    o=mozilla...THIS_IS_SDPARTA-75.0 2487803064638739471 1 IN IP4 0.0.0.0
    s=-
    t=0 0
    a=sendrecv
    a=fingerprint:sha-256 91:36:AF:63:EF:86:2D:1F:DB:C1:7E:E4:7A:C9:16:A4:AD:0F:D0:A5:4B:F1:08:65:36:3B:32:8A:19:B8:FD:58
    a=group:BUNDLE 0 1 2 3 4
    a=ice-options:trickle
    a=msid-semantic:WMS *
    m=video 9 UDP/TLS/RTP/SAVPF 96 98 126 97
    c=IN IP4 0.0.0.0
    a=recvonly
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
    a=extmap:5 urn:ietf:params:rtp-hdrext:toffset
    a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
    a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
    a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
    a=fmtp:96 max-fs=12288;max-fr=60
    a=fmtp:98 max-fs=12288;max-fr=60
    a=ice-pwd:e0e1a833f36d9e0fe0012984ec6dd249
    a=ice-ufrag:9d8a2bf7
    a=mid:0
    a=rtcp-fb:96 nack
    a=rtcp-fb:96 nack pli
    a=rtcp-fb:96 ccm fir
    a=rtcp-fb:96 goog-remb
    a=rtcp-fb:98 nack
    a=rtcp-fb:98 nack pli
    a=rtcp-fb:98 ccm fir
    a=rtcp-fb:98 goog-remb
    a=rtcp-fb:126 nack
    a=rtcp-fb:126 nack pli
    a=rtcp-fb:126 ccm fir
    a=rtcp-fb:126 goog-remb
    a=rtcp-fb:97 nack
    a=rtcp-fb:97 nack pli
    a=rtcp-fb:97 ccm fir
    a=rtcp-fb:97 goog-remb
    a=rtcp-mux
    a=rtpmap:96 VP8/90000
    a=rtpmap:98 VP9/90000
    a=rtpmap:126 H264/90000
    a=rtpmap:97 H264/90000
    a=setup:actpass
    a=ssrc:3368318818 cname:{569eb366-f0a8-49c2-ae9b-5093168654e2}
    m=audio 9 UDP/TLS/RTP/SAVPF 111 0 8 9 101
    c=IN IP4 0.0.0.0
    a=recvonly
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=extmap:2/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=fmtp:111 maxplaybackrate=48000;stereo=1;useinbandfec=1
    a=fmtp:101 0-15
    a=ice-pwd:e0e1a833f36d9e0fe0012984ec6dd249
    a=ice-ufrag:9d8a2bf7
    a=mid:1
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:9 G722/8000/1
    a=rtpmap:101 telephone-event/8000
    a=setup:actpass
    a=ssrc:2076023069 cname:{569eb366-f0a8-49c2-ae9b-5093168654e2}
    m=application 9 DTLS/SCTP 5000
    c=IN IP4 0.0.0.0
    a=sendrecv
    a=ice-pwd:e0e1a833f36d9e0fe0012984ec6dd249
    a=ice-ufrag:9d8a2bf7
    a=mid:2
    a=sctpmap:5000 webrtc-datachannel 256
    a=setup:actpass
    a=max-message-size:1073741823
    m=video 0 UDP/TLS/RTP/SAVPF 120 121 126 97
    c=IN IP4 0.0.0.0
    a=bundle-only
    a=sendrecv
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
    a=extmap:5 urn:ietf:params:rtp-hdrext:toffset
    a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
    a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
    a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
    a=fmtp:120 max-fs=12288;max-fr=60
    a=fmtp:121 max-fs=12288;max-fr=60
    a=ice-pwd:e0e1a833f36d9e0fe0012984ec6dd249
    a=ice-ufrag:9d8a2bf7
    a=mid:3
    a=msid:- {db5d210c-451c-4174-bbae-da94bd1c1626}
    a=rtcp-fb:120 nack
    a=rtcp-fb:120 nack pli
    a=rtcp-fb:120 ccm fir
    a=rtcp-fb:120 goog-remb
    a=rtcp-fb:121 nack
    a=rtcp-fb:121 nack pli
    a=rtcp-fb:121 ccm fir
    a=rtcp-fb:121 goog-remb
    a=rtcp-fb:126 nack
    a=rtcp-fb:126 nack pli
    a=rtcp-fb:126 ccm fir
    a=rtcp-fb:126 goog-remb
    a=rtcp-fb:97 nack
    a=rtcp-fb:97 nack pli
    a=rtcp-fb:97 ccm fir
    a=rtcp-fb:97 goog-remb
    a=rtcp-mux
    a=rtpmap:120 VP8/90000
    a=rtpmap:121 VP9/90000
    a=rtpmap:126 H264/90000
    a=rtpmap:97 H264/90000
    a=setup:actpass
    a=ssrc:1210881150 cname:{569eb366-f0a8-49c2-ae9b-5093168654e2}
    m=audio 0 UDP/TLS/RTP/SAVPF 109 9 0 8 101
    c=IN IP4 0.0.0.0
    a=bundle-only
    a=sendrecv
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=extmap:2/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
    a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
    a=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1
    a=fmtp:101 0-15
    a=ice-pwd:e0e1a833f36d9e0fe0012984ec6dd249
    a=ice-ufrag:9d8a2bf7
    a=mid:4
    a=msid:- {e0e11a51-926c-4e84-803f-8a1f46be31b7}
    a=rtcp-mux
    a=rtpmap:109 opus/48000/2
    a=rtpmap:9 G722/8000/1
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:101 telephone-event/8000
    a=setup:actpass
    a=ssrc:1103474835 cname:{569eb366-f0a8-49c2-ae9b-5093168654e2}
    
    
    Remote SDP (Answer)
    v=0
    o=- 821757822 1588020760 IN IP4 0.0.0.0
    s=-
    t=0 0
    a=sendrecv
    a=fingerprint:sha-256 56:52:08:AD:85:F0:F2:04:BE:05:17:F8:9A:75:B5:65:31:1A:C1:B2:68:90:23:5F:AA:87:05:2F:DA:F9:3D:09
    a=group:BUNDLE 0 1 2
    m=video 9 UDP/TLS/RTP/SAVPF 96 98 102
    c=IN IP4 0.0.0.0
    a=candidate:foundation 1 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 1 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=candidate:foundation 2 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.234.33 63191 typ host
    a=candidate:foundation 1 udp 2130706431 192.168.1.121 63192 typ host
    a=candidate:foundation 1 udp 2130706431 192.168.56.1 63193 typ host
    a=candidate:foundation 1 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194
    a=sendrecv
    a=end-of-candidates
    a=fmtp:102 profile-level-id=42001f;level-asymmetry-allowed=1;packetization-mode=1
    a=ice-pwd:vRStOTenOAAheuTypEDtRIewUaZaLmkQ
    a=ice-ufrag:vRStOTenOAAheuTy
    a=mid:0
    a=msid:game-video video
    a=rtcp-mux
    a=rtpmap:96 VP8/90000
    a=rtpmap:98 VP9/90000
    a=rtpmap:102 H264/90000
    a=setup:actpass
    a=ssrc:271976615 cname:game-video
    a=ssrc:271976615 msid:game-video video
    a=ssrc:271976615 mslabel:game-video
    a=ssrc:271976615 label:video
    m=audio 9 UDP/TLS/RTP/SAVPF 111 0 8 9
    c=IN IP4 0.0.0.0
    a=candidate:foundation 1 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 1 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=candidate:foundation 2 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=sendrecv
    a=end-of-candidates
    a=fmtp:111 maxplaybackrate=0;stereo=0;useinbandfec=1
    a=ice-pwd:vRStOTenOAAheuTypEDtRIewUaZaLmkQ
    a=ice-ufrag:vRStOTenOAAheuTy
    a=mid:1
    a=msid:game-audio audio
    a=rtcp-mux
    a=rtpmap:111 OPUS/48000/2
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:9 G722/8000/1
    a=setup:actpass
    a=ssrc:1333919895 cname:game-audio
    a=ssrc:1333919895 msid:game-audio audio
    a=ssrc:1333919895 mslabel:game-audio
    a=ssrc:1333919895 label:audio
    m=application 9 DTLS/SCTP 5000
    c=IN IP4 0.0.0.0
    a=candidate:foundation 1 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.234.33 63191 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.1.121 63192 typ host generation 0
    a=candidate:foundation 1 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 2 udp 2130706431 192.168.56.1 63193 typ host generation 0
    a=candidate:foundation 1 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=candidate:foundation 2 udp 1694498815 95.30.103.36 63194 typ srflx raddr 0.0.0.0 rport 63194 generation 0
    a=sendrecv
    a=end-of-candidates
    a=ice-pwd:vRStOTenOAAheuTypEDtRIewUaZaLmkQ
    a=ice-ufrag:vRStOTenOAAheuTy
    a=mid:2
    a=sctpmap:5000 webrtc-datachannel 1024
    a=setup:actpass
    
  • Can't start any game

    Can't start any game

    I got some error when starting the server and can't start any game on localhost:8000.

    2019/09/10 10:30:56 Listening at port: localhost:8000
    2019/09/10 10:30:56 Running as worker
    2019/09/10 10:30:56 Warn: Failed to create client: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    2019/09/10 10:30:56 Listening at port: localhost: 9000
    .....
    .....
    2019/09/10 10:31:04 === StartClient ===
    2019/09/10 10:31:04 Start peerconnection 1b9e405b-1c5f-4e67-84a8-26d76d84d7de
    2019/09/10 10:31:04 Error: Cannot create new webrtc session mDNS: failed to join multicast group
    2019/09/10 10:31:04 Overlord: Received sdp request from a worker
    2019/09/10 10:31:04 Overlord: Sending back sdp to browser
    Connected
    
  • Very slow video performance

    Very slow video performance

    Hi,

    I succesfully installed in my own VPS a version of cloud-game.

    From a functional point of view, everything seems to work right, but the performance of the videostream makes the emulation unplayable.

    Audio is smooth and responsive, but video halts constantly. Video works during 2-3 seconds, halts but audio and emulation goes ahead, Video catchs ups after severe frameskipping.

    I've tried cloudretro.io and work great, with the same game (NES Contra in this case).

    Some extra info:

    C:\Users\anton>ping cloudretro.io
    
    Haciendo ping a cloudretro.io [167.172.70.98] con 32 bytes de datos:
    Respuesta desde 167.172.70.98: bytes=32 tiempo=167ms TTL=47
    Respuesta desde 167.172.70.98: bytes=32 tiempo=167ms TTL=47
    Respuesta desde 167.172.70.98: bytes=32 tiempo=166ms TTL=47
    Respuesta desde 167.172.70.98: bytes=32 tiempo=168ms TTL=47
    
    Estadísticas de ping para 167.172.70.98:
        Paquetes: enviados = 4, recibidos = 4, perdidos = 0
        (0% perdidos),
    Tiempos aproximados de ida y vuelta en milisegundos:
        **Mínimo = 166ms, Máximo = 168ms, Media = 167ms**
    
    C:\Users\anton>ping 161.22.43.92 (-----> my VPS)
    
    Haciendo ping a 161.22.43.92 con 32 bytes de datos:
    Respuesta desde 161.22.43.92: bytes=32 tiempo=33ms TTL=57
    Respuesta desde 161.22.43.92: bytes=32 tiempo=38ms TTL=57
    Respuesta desde 161.22.43.92: bytes=32 tiempo=32ms TTL=57
    Respuesta desde 161.22.43.92: bytes=32 tiempo=32ms TTL=57
    
    Estadísticas de ping para 161.22.43.92:
        Paquetes: enviados = 4, recibidos = 4, perdidos = 0
        (0% perdidos),
    Tiempos aproximados de ida y vuelta en milisegundos:
        **Mínimo = 32ms, Máximo = 38ms, Media = 33ms**
    
    

    my VPS seems to deal with things properly:

    • 2 vCORES using 15% of capacity in worst scenario
    • 800 kbps on network bandwith consumption when playing

    Any clue on where to try to find the source of my problem?

    Kind Regards

  • using cpu capabilities: none!   --- H264 performance issue

    using cpu capabilities: none! --- H264 performance issue

    hi, when I was using h264 to encode the video, I can see the video begin to stuck.

    the logs shows like this: I0202 16:25:33.807307 1926 media.go:134] Video Encoder: H264 x264 [info]: using cpu capabilities: none! x264 [info]: profile Constrained Baseline, level 5.0

    is the performence issue caused by this: using cpu capabilities: none!

  • Use modified x264 lib

    Use modified x264 lib

    x264 info

    Settings

    A list of useful settings extracted from the source x264c.

    • profile Force the limits of an H.264 profile. Overrides all settings. (baseline, main, high, ...)
    for X264_CSP_I420 image encoding
    
       - baseline:
            --no-8x8dct --bframes 0 --no-cabac --cqm flat --weightp 0
            No interlaced.
            No lossless.
       - main:
            --no-8x8dct --cqm flat
            No lossless.
       - high:
            No lossless.
    
    • preset Use a preset to select encoding settings [medium]. Overridden by user settings. (ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow, placebo)
        - ultrafast:
            --no-8x8dct --aq-mode 0 --b-adapt 0 --bframes 0 --no-cabac --no-deblock
            --no-mbtree --me dia --no-mixed-refs --partitions none --rc-lookahead 0 --ref 1
            --scenecut 0 --subme 0 --trellis 0 --no-weightb --weightp 0
    
        - superfast:
            --no-mbtree --me dia --no-mixed-refs --partitions i8x8,i4x4 --rc-lookahead 0
            --ref 1 --subme 1 --trellis 0 --weightp 1
    
        - veryfast:
            --no-mixed-refs --rc-lookahead 10 --ref 1 --subme 2 --trellis 0 --weightp 1
    
        - faster:
            --no-mixed-refs --rc-lookahead 20 --ref 2 --subme 4 --weightp 1
    
        - fast:
            --rc-lookahead 30 --ref 2 --subme 6 --weightp 1
    
        - medium (Default settings apply)
     
        - slow:
            --direct auto --rc-lookahead 50 --ref 5 --subme 8 --trellis 2
    
        - slower:
            --b-adapt 2 --direct auto --me umh --partitions all --rc-lookahead 60 --ref 8 --subme 9 --trellis 2
    
        - veryslow:
            --b-adapt 2 --bframes 8 --direct auto --me umh --merange 24 --partitions all --ref 16 
            --subme 10 --trellis 2 --rc-lookahead 60
    
         - placebo:
            --bframes 16 --b-adapt 2 --direct auto --slow-firstpass --no-fast-pskip
            --me tesa --merange 24 --partitions all --rc-lookahead 60 --ref 16 --subme 11 --trellis 2
    
    • tune Tune the settings for a particular type of source or situation. Overridden by user settings. Multiple tunings are separated by commas. Only one psy tuning can be used at a time. (psy tunings: film, animation, grain, stillimage, psnr, ssim / other tunings: fastdecode, zerolatency)
        - film (psy tuning):
            --deblock -1:-1 --psy-rd <unset>:0.15
    
        - animation (psy tuning):
            --bframes {+2} --deblock 1:1 --psy-rd 0.4:<unset> --aq-strength 0.6 --ref {Double if >1 else 1}
    
        - grain (psy tuning):
            --aq-strength 0.5 --no-dct-decimate --deadzone-inter 6 --deadzone-intra 6
            --deblock -2:-2 --ipratio 1.1--pbratio 1.1 --psy-rd <unset>:0.25 --qcomp 0.8
    
        - stillimage (psy tuning):
            --aq-strength 1.2 --deblock -3:-3 --psy-rd 2.0:0.7 
    
        - psnr (psy tuning):
            --aq-mode 0 --no-psy
    
        - ssim (psy tuning):
            --aq-mode 2 --no-psy
    
        - fastdecode:
            --no-cabac --no-deblock --no-weightb --weightp 0
    
        - zerolatency:
            --bframes 0 --force-cfr --no-mbtree --sync-lookahead 0 --sliced-threads --rc-lookahead 0
    

    Changes

    • The old h264 encoder is rewritten in order to be able to make use of the external libx264 system library with SIMD instructions.
    • Huge memory leaks are fixed.
    • Added use of C YUV-converter.
    • Added ability to change video codec and some (h264) config params in the settings file.

    Issues

    • It is possible that libx264 support in the app will be broken again after another API (C header) change by its authors.
    • h264 encoder is harder to tweak for real-time encoding compared to more advanced and new vpx encoders.
  • Fix errors/misuse with OpenGL-based core API

    Fix errors/misuse with OpenGL-based core API

    This PR is dedicated to fixing some errors and misuses in the OpenGL HW-accelerated Libretro cores renderer.

    How Libretro cores utilize OpenGL.

    If a Libretro core needs to use OpenGL, for example, to emulate 3D-accelerated games, in that case, your Libretro frontend has to own and share the OpenGL context with the core. The only job for the core is to render graphics for the next frame and put the final result into a shared FrameBufferObject (FBO) [3], all that done with a special render-to-texture OpenGL extension that should be supported by the video driver (OpenGL 3+). After that, your frontend can read the image from the FBO and show it in any convenient way.

    A simplified list of steps to deal with OpenGL Libretro cores:

    • Initialize the OpenGL context.
      • Make a window with the help of SDL. And we will need a window because OpenGL context initialization needs to gather data from the windowing system even though the rendering is going to be off-screen. (perhaps it is possible to reuse an existing window in the OS or even destroy new window right after context creation but this idea requires verification) ¯\_(ツ)_/¯
      • Using some OpenGL Loading Library (e.g. go-gl [Glow]), load pointers to OpenGL functions at runtime, core, and extensions.
      • Call the OpenGL init function and that will be enough.
      • Don't forget that before calling any GL functions in a thread, context should be made current (thread-local variable). That way each thread gonna have its own copy of the OpenGL context state to change and should flush these changes during your next context-make-current calls or will keep them inside the thread otherwise [4].
    • Initialize FrameBufferObject and a texture to draw on.
      • By default windowing system provide you with its own framebuffer but we need to create one explicitly in order to minimize data copy and increase the performance for core off-screen rendering.
      • To let OpenGL perform rendering into a texture, an image of a texture object must be attached to a framebuffer (GL_COLOR_ATTACHMENT).
      • Additionally, you can specify the depth and stencil components of the framebuffer (all three also are called attachments) if the core returns info about the usage of those.
      • Bind FBO's name to the current context for reading/writing operation allowance with GL_FRAMEBUFFER param (I'm not sure about the need for this step since it makes no sense here without any data to read, I yet to know how to properly bind/unbind that buffer between frontend/core, especially with an amazing ability of go-routines to jump between threads).
    • Notify the core that you are done (call context_reset callback).
    • Grab the image from the FBO in the video_refresh callback when it's ready.
      • Check the RETRO_HW_FRAMEBUFFER_VALID flag as an indicator for HW rendering.
      • Bind the framebuffer for reading op.
      • Read pixels from it (glReadPixels or glTexSubImage2D).

    Sources:

    1. Hans-Kristian Arntzen Implementing a Hardware Accelerated Libretro Core (pdf)
    2. Interactive Computer Graphics. 6.1 OpenGL Version, Context, and Profile (link)
    3. songho.ca OpenGL (link)
    4. WinAPI wglMakeCurrent (link)
    5. The Red Book (:<

    Issues:

    • SDL2 is a pretty bloated framework just for window creation, GLFW is much lighter
    • [x] to check OpenGL Cores Libretro doc
    Video driver should support render-to-texture extension (OpenGL 3.0+ Core).
    
    • [x] unnecessary OpenGL 4.1 Core profile use, it's better to follow 2.1 profile
    Switched to OpenGL 2.1 bindings.
    
    • [x] OpenGL coordinate system is flipped, use of slow and unnecessary third-party dependency just for the image vertical flip was unjustified

    https://github.com/giongto35/cloud-game/blob/a42aebec886f479be43ad33661c43077085b1c6b/pkg/emulator/libretro/nanoarch/nanoarch.go#L153-L166

    1. OpenGL's Lower Left Origin

    Given a sheet of paper, people write from the top of the page to the bottom. The origin for writing text is at the upper left-hand margin of the page (at least in European languages). However, if you were to ask any decent math student to plot a few points on an X-Y graph, the origin would certainly be at the lower left-hand corner of the graph. Most 2D rendering APIs mimic writers and use a 2D coordinate system where the origin is in the upper left-hand corner of the screen or window (at least by default). On the other hand, 3D rendering APIs adopt the mathematically minded convention and assume a lower left-hand origin for their 3D coordinate systems.

    If you are used to 2D graphics APIs, this difference of origin location can trip you up. When you specify 2D coordinates in OpenGL, they are generally based on a lower left-hand coordinate system. Keep this in mind when using glViewport, glScissor, glRasterPos2i, glBitmap, glTexCoord2f, glReadPixels, glCopyPixels, glCopyTexImage2D, glCopyTexSubImage2D, gluOrtho2D, and related routines.

    Another common pitfall related to 2D rendering APIs having an upper left-hand coordinate system is that 2D image file formats start the image at the top scan line, not the bottom scan line. OpenGL assumes images start at the bottom scan line by default. If you do need to flip an image when rendering, you can use glPixelZoom(1,-1) to flip the image in the Y direction. Note that you can also flip the image in the X direction. Figure 4 demonstrates using glPixelZoom to flip an image.

    Note that glPixelZoom only works when rasterizing image rectangles with glDrawPixels or glCopyPixels. It does not work with glBitmap or glReadPixels. Unfortunately, OpenGL does not provide an efficient way to read an image from the frame buffer into memory starting with the top scan line.

    Rewrite it either similarly to rotationFn aka 0-cost frame slice reader (+ combine with rotations somehow) or use GL matrix transform (which is slower):

    Libretro treats top-left as origin. OpenGL treats bottom-left as origin. To be compatible with the libretro model, top-left semantics are preserved. Rendering normally will cause the image to be flipped vertically. To avoid this, simply scale the final projection matrix by [1,− 1 , 1 ,1].

    Added a function that flips Y coordinate for each pixel after rotation and before RGBA image draw.
    
    • [x] Pixel formats can change from core to core, misused for textures and the frame buffer (BRG/RGB)
    Used corresponding OpenGL pixel formats and types based on core callback values.
    
    • FBO read speed can be increased with PBO.

    • [x] macOS "main thread problem" solution causes a crash with the native Windows build of the app (https://github.com/golang/go/wiki/LockOSThread)

    From go-sdl2 FAQ:

    Why does my program crash randomly or hang? Putting runtime.LockOSThread() at the start of your main() usually solves the problem (see SDL2 FAQ about multi-threading).

    UPDATE: Recent update added a call queue system where you can put thread-sensitive code and have it called synchronously on the same OS thread. See the render_queue or render_goroutines examples from https://github.com/veandco/go-sdl2-examples to see how it works.

    Added SDL/GL functions into main.thread wrapper. (rewrite later: remove mainthread lib 
    in favor of go-sdl2's included render-queue)
    

    image

    Seems that some problem with retro_load_game / retro_unload_game function in the worker.
    SDL/OpenGL deinitialization broken at least in Windows, a bunch of zombie? threads remain.
    Postponed until backporting normal test runners for Libretro cores.
    
    [won't fix] RetroArch with the same core has this issue. 
    
    • [x] to try to check strangely deterministic audio stutter with the native Windows build :( WebRTC audio jitter all over the place image
    A possible cause is in the Mesa drivers (slow software llvmpipe renderer).
    Disabling OpenGL drawing in the core removes audio stuttering.
    
  • having trouble running N64 on ubuntu, No available video device

    having trouble running N64 on ubuntu, No available video device

    Am I missing a video driver? Other emulators work. Thanks


    --- System audio and video info ---

    Aspect ratio: 1.3333333730697632 Base width: 320 Base height: 240 Max width: 320 Max height: 240 Sample rate: 44100 FPS: 60

    panic:

    goroutine 17 [running, locked to thread]: github.com/giongto35/cloud-game/pkg/emulator/libretro/nanoarch.initVideo() /root/cloud-game/pkg/emulator/libretro/nanoarch/nanoarch.go:376 +0x993 github.com/giongto35/cloud-game/pkg/emulator/libretro/nanoarch._cgoexpwrap_58c817539c33_initVideo() _cgo_gotypes.go:752 +0x25 make: *** [Makefile:77: dev.run] Error 2

  • N64 Support

    N64 Support

    Add support for OpenGL cores in general but only tested with Mupen64plus and Mupen64plusNext on a Raspberry Pi. It probably needs more work to run on every system and with other OpenGL libretro libraries. Also adds support for analog stick from frontend to backend. Also adds CLI settings that allow to run coordinator and worker over HTTPS on different setups.

  • Support function parameters when bypassing LibCo cores

    Support function parameters when bypassing LibCo cores

    Some Libretro cores that use LibCo (C coprocessing library) don't work well with Go multithreading model. The current implementation with the hard serialization on the C side doesn't take into account external function calls that have non-void parameters. So these function calls will crash emulators during concurrent memory access.

        In the meantime I tried a super ugly hack in order to test my theory and it seems I was right. Now I know what problem is, but not sure how to fix it properly (C is not my strong suit and I'm not the author of the N64 support here). Maybe, I will fix it some time later.
    

    https://user-images.githubusercontent.com/846874/206714612-6f68ec84-1237-438b-aff3-735fa6f18866.mp4

    Originally posted by @sergystepanov in https://github.com/giongto35/cloud-game/issues/384#issuecomment-1344322888

  • controller support (game controller mode) for edge(chromium) on xbox one/series consoles

    controller support (game controller mode) for edge(chromium) on xbox one/series consoles

    I can access a cloudretro app on my xbox one x, but the controller does not attach exclusively to the game by default. (input is also interpreted as mouse movement within the browser window.)

    This can be fixed by switching to "game controller mode" within the browser (pressing the hamburger button on the controller and selecting "game controller mode") It would be nice to have this enabled by default, since the intention of this app is to play a game.

    This causes the input to continue to be used as navigation for the web browser. (e.g. "B" load the previous page).

    This obviously makes the cloud-game application cumbersome to use on an xbox.

    If i'm not mistaken this is the documentation required to correctly implement xbox controller input scheme on xbox edge:

    https://docs.microsoft.com/en-us/windows/apps/design/input/gamepad-and-remote-interactions#xy-focus-navigation-and-interaction

    Sadly I lack the technical know how to do this myself, so I submit this as a feature request.

  • dosbox_pure_libretro.so keyboard mapping

    dosbox_pure_libretro.so keyboard mapping

    my end goal is to get dosbox games running in the same way that the other cores work.

    I successfully installed dosbox_pure_libretro.so as a new core and added a game to the games folder adjusted config.yaml, started up the server and worker and got a game running.

    however it seems the keyboard does not map to what the dosbox core expects.

    imho I would expect that web/js/input/input.js and web/js/input/keyboard.js have a role to play in the client side of the controller support.

    no clue about the server side though.

  • Refactored v3

    Refactored v3

    This PR contains refactored code.

    Changelog

    • Added new net code (the communication architecture was left intact).
    • All network client IDs now have custom type network.Uid backed by github.com/rs/xid lib.
      The string representation of a UUID takes 32 bytes, and the new type will take just 16.
      Because of Golang JSON serialization problems with omitting zero-length empty slices (it can't) 
      and the need to use UID values as map keys (maps don't support slices as keys), 
      IDs are stored as strings (for now).
      
    • A whole new WebSocket client/server implementation was added, as well as a new communication layer with synchronous and async call handlers.
      • WebSocket connections now support dedicated Ping/Pong frames as opposed to original ping text messages.
      • Used Gorilla WebSocket library doesn't allow concurrent (simultaneous) reads and writes, so this part was handled via send channel synchronization.
    • New API structures can be found in the pkg/api folder.
    • New communication protocol can be found in the pkg/com/* folder.
    • Updated communication protocol is based on JSON-encoded messaging through WebSocket and has the following structure:
      Packet
        [id] string — a globally unique identification tag for the packet to track it trough a chain of requests.
        t uint8 — contains packet type information (i.e. INIT_PACKET, SDP_OFFER_PACKET, ...).
        [p] interface{} — contains packet data (any type).
      
      Each packet is a text message in JSON-serialized form (WebSocket control frames obviously not).
      
      The main principle of this protocol and the duplex data exchange is:
      the one who initializes connection is called a client, and 
      the one who is being connected to is called a server. 
      With the current architecture, the coordinator is the server, the user browsers and workers are the clients.
      
                ____           ____
               ↓    ↑         ↑    ↓
         browser ⟶ coordinator ⟵ worker
           (c)          (s)         (c)
      
      One of the most crucial performance vise parts of these interactions is that 
      all the server-initiated calls to clients should be asynchronous!
      
      • In order to track synchronous calls (packets) with an asynchronous protocol, such as WebSocket, each packet may have an id that should be copied in all subsequent requests/responses.
      • The old sessionID param was replaced by id that should be stored inside the p (payload) part of the packet.
    • It is possible to skip the default ping check for all connected workers on every user connection and just pick the first available with the new roundRobin param in the coordinator config file coordinator.roundRobin: true/false.
    • Added a dedicated package for the system API (pkg/api/*).
    • Added structured logging system (zerolog) for better logging and cloud services integration.
    • Added a visual representation of the network message exchange in logs:
      ...
      01:00:01.1078 3f98 INF w → c Handshake ws://localhost:8000/wso
      01:00:01.1138  994 INF c ← w Handshake localhost:8000
      01:00:01.1148  994 INF c ← w Connect cid=cep.hrg
      01:00:01.1158  994 DBG c     connection id has been changed to cepl7obdrc3jv66kp2ug cid=cep.hrg
      01:00:01.1158 3f98 INF w → c Connect cid=cep.2ug
      01:00:01.1158  994 INF c     New worker / addr: localhost, ...
      01:00:01.1158 3f98 INF w     Connected to the coordinator localhost:8000 cid=cep.2ug
      01:00:02.5834  994 INF c ← u Handshake localhost:8000
      01:00:02.6175  994 INF c ← u Connect cid=cep.hs0
      01:00:02.6209  994 INF c     Search available workers cid=cep.hs0
      01:00:02.6214  994 INF c     Found next free worker cid=cep.hs0
      01:00:02.6220  994 INF c → u InitSession cid=cep.hs0
      01:00:02.6527  994 INF c ← u WebrtcInit cid=cep.hs0
      01:00:02.6527  994 INF c → w ᵇWebrtcInit cid=cep.hrg
      01:00:02.6537 3f98 INF w ← c WebrtcInit cid=cep.2ug
      01:00:02.6537 3f98 INF w     WebRTC start cid=cep.2ug
      ...
      
    • Replaced a monstrous Prometheus metrics lib.
    • Removed spflag dependency.
    • Added new version config file param/constant for compatibility reasons.
    • Bump the minimum required version for Go to 1.18 due to use of generics.
    • Opus encoder now is cached and the default config is 96Kbps, complexity 5 (was 196Kbps, 8).
    • Changed the default x264 quality parameters to crf 23 / superfast / baseline instead of crf 17 / veryfast / main.
    • Added a separate WebRTC logging config param webrtc.logLevel.
    • Worker now allocates much less memory.
    • Optimized RGB to YUV converter.

    Breaking changes (migration to v3)

    • Coordinator server API changes, see web/js/api/api.js.
    • Coordinator client event API changes:
      • c GAME_PLAYER_IDX_CHANGE (string) -> GAME_PLAYER_IDX (number)
      • c GAME_PLAYER_IDX -> GAME_PLAYER_IDX_SET
      • c MEDIA_STREAM_INITIALIZED -> WEBRTC_NEW_CONNECTION
      • c MEDIA_STREAM_SDP_AVAILABLE -> WEBRTC_SDP_OFFER
      • c MEDIA_STREAM_CANDIDATE_ADD -> WEBRTC_ICE_CANDIDATE_RECEIVED
      • c MEDIA_STREAM_CANDIDATE_FLUSH -> WEBRTC_ICE_CANDIDATES_FLUSH
      • x MEDIA_STREAM_READY -> removed
      • c CONNECTION_READY -> WEBRTC_CONNECTION_READY
      • c CONNECTION_CLOSED -> WEBRTC_CONNECTION_CLOSED
      • c GET_SERVER_LIST -> WORKER_LIST_FETCHED
      • x KEY_STATE_UPDATED -> removed
      • n WEBRTC_ICE_CANDIDATE_FOUND
      • n WEBRTC_SDP_ANSWER
      • n MESSAGE
    • rtcp module renamed to webrtc.
    • Controller state equals Libretro controller state (changed order of bits), see: web/js/input/input.js.
    • RoundRobin config param enabled by default (selects any free worker instead of ping). Set this param to false for the old behavior.
    • --v=5 logging cmd flag was removed and replaced with the debug config parameter.
    • Changed the name of the webrtc.iceServers.url config param to webrtc.iceServers.urls.
    • Removed unused YUV mode for 1px color chroma select.
    • Fixed rounding error for proper YUV color calculations.
  • Joystick key rebinding support

    Joystick key rebinding support

    emmm, is it possible to support joystick rebinding in the future? i need it..

    Originally posted by @xxxsen in https://github.com/giongto35/cloud-game/issues/338#issuecomment-890310768

Gamespictionary - Scribble.rs is an alternative to the web-based drawing game skribbl.io

Scribble.rs Scribble.rs is an alternative to the web-based drawing game skribbl.

Apr 13, 2022
Arkanoid game in Go using Ebiten game engine with ECS.
Arkanoid game in Go using Ebiten game engine with ECS.

Arkanoid-go Arkanoid game in Go using Ebiten game engine with ECS. You must have Git LFS installed when cloning the repository to download assets. See

Oct 9, 2022
AircraftWar - a game powered by Go+ spx game engine
AircraftWar - a game powered by Go+ spx game engine

AircraftWar - a game powered by Go+ spx game engine How to run Download Go+ and build it. See https://github.com/goplus/gop#how-to-build. Download thi

Jan 5, 2022
FlappyCalf - a game powered by Go+ spx game engine
FlappyCalf - a game powered by Go+ spx game engine

FlappyCalf - a game powered by Go+ spx game engine How to run Download Go+ and build it. See https://github.com/goplus/gop#how-to-build. Download this

Nov 6, 2022
FlappyCalf - a game powered by Go+ spx game engine
FlappyCalf - a game powered by Go+ spx game engine

FlappyCalf - a game powered by Go+ spx game engine How to run Download Go+ and build it. See https://github.com/goplus/gop#how-to-build. Download this

Nov 6, 2022
MazePlay - a game powered by Go+ spx game engine
MazePlay - a game powered by Go+ spx game engine

MazePlay - a game powered by Go+ spx game engine How to run Download Go+ and build it. See https://github.com/goplus/gop#how-to-build. Download this g

Dec 16, 2021
A simple game that I created with Ebiten game library as a way to teach myself Go. Enjoy!
A simple game that I created with Ebiten game library as a way to teach myself Go. Enjoy!

galactic-asteroid-belt Overview A simple game that I created with Ebiten game library as a way to teach myself Go. Enjoy! Run To run, you will need Go

Dec 2, 2021
RundQuiz-Game - This is a Go exercise that implements and builds a quiz game from a list of math questions in a CSV file.

Go RundQuiz Game Exercise details This exercise is broken into two parts to help simplify the process of explaining it as well as to make it easier to

Jan 5, 2022
Simple 2D game to teach myself various things about game development and ECS, etc

2d-grass-game I really don't know what to name this game. Its a big grass field, and its in 2d so....2D Grass game This is a simple 2D game to teach m

Jan 17, 2022
A game server side framework with both web API and realtime communication.

HAYABUSA Framework Hayabusa is a server side framework for Japan-like social games. Easy to understand and use for beginners Powerful controller, flex

May 21, 2022
Lightweight, facility, high performance golang based game server framework
Lightweight, facility, high performance golang based game server framework

Nano Nano is an easy to use, fast, lightweight game server networking library for Go. It provides a core network architecture and a series of tools an

Jan 1, 2023
Terminal-based game engine for Go, built on top of Termbox
Terminal-based game engine for Go, built on top of Termbox

Termloop Termloop is a pure Go game engine for the terminal, built on top of the excellent Termbox. It provides a simple render loop for building game

Dec 29, 2022
Simple rule based matchmaking for your online game with support of Redcon(RESP) protocol.
 Simple rule based matchmaking for your online game with support of Redcon(RESP) protocol.

Simple Matchmaking Simple rule based matchmaking for your online game with support of Redcon(RESP) protocol. 1- Simple Match Rule Easiest usage of sys

Jan 4, 2023
Go-based 2D game to relax after a long coding time

MoreLoves As simple 2D game to play during time when you need to relax after a deep coding time. On your lovely terminal, use your keyboard keys to na

May 21, 2022
Tcell-game-template - A small template repository for simple tcell based games

tcell game template This is a template repository used for making small terminal

Jan 22, 2022
Librebird - A text based bird game made in Go

librebird a text based bird game made in Go, please contribute to the adventures

Feb 19, 2022
A dead simple 2D game library for Go
A dead simple 2D game library for Go

Ebiten (v2) A dead simple 2D game library for Go Ebiten is an open source game library for the Go programming language. Ebiten's simple API allows you

Dec 28, 2022
Engo is an open-source 2D game engine written in Go.

Engo A cross-platform game engine written in Go following an interpretation of the Entity Component System paradigm. Engo is currently compilable for

Dec 26, 2022
Go 3D Game Engine
Go 3D Game Engine

G3N - Go 3D Game Engine G3N (pronounced "gen") is an OpenGL 3D Game Engine written in Go. It can be used to write cross-platform Go applications that

Jan 9, 2023